Versions Compared

Old Version 2

changes.mady.by.user Νικολαος Κωστοπουλος

Saved on

compared with

New Version 3

changes.mady.by.user Νικολαος Κωστοπουλος

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

...

The following steps apply for both installation options. WiFiMon users who will use the prepared WHP image (installation option 1) should simply edit the crontab and wireless.py and twping_parser.py files as discussed in the following. WiFiMon users who will not use the prepared WIFiMon WHP image (installation option 2) should follow the steps 2 up to 45.

Step 1: Write the image to the micro SD card

Follow the instructions at the official Raspberry Pi site. Skip the "Download the image" step and use the WiFiMon Raspberry Pi operating system image instead (download size is approx. 8 3 GB).

WiFiMon Raspberry Pi image given above is a custom version of Raspberry Pi OS (Buster) with desktop, with the default Raspberry Pi credentials (user: pi, password: raspberry).

...

Code Block
languagebash
00,10,20,30,40,50 * * * * Xvfb :100 &
02,12,22,32,42,52 * * * * export DISPLAY=:100 && firefox-esr --new-tab URL_TO_nettest.html >/dev/null 2>&1
04,14,24,34,44,54 * * * * export DISPLAY=:100 && firefox-esr --new-tab URL_TO_speedworker.html >/dev/null 2>&1
06,16,26,36,46,56 * * * * export DISPLAY=:100 && firefox-esr --new-tab URL_TO_boomerang.html >/dev/null 2>&1
08,18,28,38,48,58 * * * * /home/pi/wireless.py >> ~/cron.log 2>&1
05,15,25,35,45,55 * * * * /home/pi/twping_parser.py >> ~/cron.log 2>&1

You have to modify the following parts of the crontab in lines 2-4:

...

Line 5 of the crontab is related to the streaming of wireless network interface metrics to the WiFiMon Analysis Server (WAS). Optionally, the intervals of the WHP measurements could be altered by appropriately configuring the crontab so that measurement are more or less frequent. The configuration of the crontab config given above sets up 10-minute intervals between the measurements of each test tool in a way in which there are no overlapping measurements.

Line 6 of the crontab is related to the streaming of TWAMP measurement results to the WiFiMon Analysis Server (WAS).

Step 4: Streaming Wireless Network Interface Metrics to the WiFiMon Analysis Server (WAS)
Anchor
Step4
Step4

...

Code Block
languagepy
firstline1
titlewireless.py
linenumberstrue
#!/usr/bin/python3

import subprocess
import datetime
import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
import json

def return_command_output(command):
    proc = subprocess.Popen(command, stdout = subprocess.PIPE, shell = True)
    (out, err) = proc.communicate()
    output = out.rstrip('\n'.encode('utf8'))
    return output

def get_mac(iface):
    command = "cat /sys/class/net/" + str(iface) + "/address"
    mac = return_command_output(command).decode('utf8')
    mac = mac.replace(":", "-")
    return mac

def find_wlan_iface_name():
    command = "printf '%s\n' /sys/class/net/*/wireless | awk -F'/' '{print $5 }'"
    wlan_iface_name = return_command_output(command)
    return wlan_iface_name.decode('utf8')

def parse_iwconfig(iface):
    bit_rate = return_command_output("sudo iwconfig " + iface + " | grep Bit | awk '{print $2}' | sed 's/Rate=//'").decode('utf8')
    tx_power = return_command_output("sudo iwconfig " + iface + " | grep Bit | awk '{print $4}' | sed 's/Tx-Power=//'").decode('utf8')
    link_quality = return_command_output("sudo iwconfig " + iface + " | grep Link | awk '{print $2}' | sed 's/Quality=//'").decode('utf8')
    link_quality = link_quality.split("/")[0]
    signal_level = return_command_output("sudo iwconfig " + iface + " | grep Link | awk '{print $4}' | sed 's/level=//'").decode('utf8')
    accesspoint = return_command_output("sudo iwconfig " + iface + " | grep Mode | awk '{print $6}' | sed 's/Point: //'").decode('utf8')
    accesspoint = accesspoint.replace(":", "-")
    essid = return_command_output("sudo iwconfig " + iface + " | grep ESSID | awk '{print $4}' | sed 's/ESSID://'").decode('utf8')
    essid = essid.replace("\"", "")
    return bit_rate, tx_power, link_quality, signal_level, accesspoint, essid

def parse_iwlist(iface, accesspoint):
    information = {}
    command = "sudo iwlist " + iface + " scan | grep -E \"Cell|Frequency|Quality|ESSID\""
    aps = return_command_output(command).decode("utf8")
    aps = aps.split("\n")

    cell_indices = list()
    for index in range(0, len(aps)):
        line_no_whitespace = ' '.join(aps[index].split())
        parts = line_no_whitespace.split()
        if parts[0] == "Cell":
            cell_indices.append(index)

    for index in cell_indices:
        line0 = ' '.join(aps[index].split())
        ap_mac = line0.split()[-1]
        ap_mac = ap_mac.replace(":", "-")
        information[ap_mac] = {}
        line1 = ' '.join(aps[index + 1].split())
        partsfrequency = line1.split()[0].split(":")[1]
        information[ap_mac]["drillTestfrequency"] = float(parts[2].split("=")[1]str(frequency)
        line2 = ' '.join(aps[index + 2].split())
        parts = line2.split(":")
        information[ap_mac][str"drillTest"] = float(parts[12].replacesplit('"', ''))] = information[ap_mac]["drillTest"]

"=")[1])
      return information

def convert_info_to_json(accesspoint, essid, mac, bit_rate, tx_power, link_quality, signal_level, probe_no, information, location_name, test_device_location_description, nat_network): line3 = ' '.join(aps[index + 3].split())
        parts = line3.split(":")
    overall_dictionary = {}
    overall_dictionary["macAddress"information[ap_mac][str(parts[1].replace('"', ''))] = "\"" + str(mac) + "\""
    overall_dictionary["accesspointinformation[ap_mac]["drillTest"]

    return information

def convert_info_to_json(accesspoint, essid, mac, bit_rate, tx_power, link_quality, signal_level, probe_no, information, location_name, test_device_location_description, nat_network, system_dictionary):
    overall_dictionary = {}
    overall_dictionary["macAddress"] = "\"" + str(accesspointmac) + "\""
    overall_dictionary["essidaccesspoint"] = "\"" + str(essid) + "\""
    bit_rate = int(float(bit_rate))
    overall_dictionary["bitRate"] = str(bit_rate)
    tx_power = int(float(tx_power))
    overall_dictionary["txPower"] = str(tx_power)
    link_quality = int(float(link_quality))
    overall_dictionary["linkQuality"] = str(link_quality)
    signal_level = int(float(signal_level))
    overall_dictionary["signalLevel"] = str(signal_level)
    overall_dictionary["probeNo"] = str(probe_no)
    information = json.dumps(information)
    overall_dictionary["monitor"] = information
    overall_dictionary["locationName"] = "\"" + str(location_name) + "\""
    overall_dictionary["testDeviceLocationDescription"] = "\"" + str(test_device_location_description) + "\""
    overall_dictionary["nat"] = "\"" + str(nat_network) + "\""
    json_data = json.dumps(overall_dictionaryaccesspoint) + "\""
    overall_dictionary["essid"] = "\"" + str(essid) + "\""
    bit_rate = int(float(bit_rate))
    overall_dictionary["bitRate"] = str(bit_rate)
    tx_power = int(float(tx_power))
    overall_dictionary["txPower"] = str(tx_power)
    link_quality = int(float(link_quality))
    overall_dictionary["linkQuality"] = str(link_quality)
    signal_level = int(float(signal_level))
    overall_dictionary["signalLevel"] = str(signal_level)
    overall_dictionary["probeNo"] = str(probe_no)
    information = json.dumps(information)
    overall_dictionary["monitor"] = information
    overall_dictionary["locationName"] = "\"" + str(location_name) + "\""
    overall_dictionary["testDeviceLocationDescription"] = "\"" + str(test_device_location_description) + "\""
    overall_dictionary["nat"] = "\"" + str(nat_network) + "\""
    system_dictionary = json.dumps(system_dictionary)
    overall_dictionary["system"] = system_dictionary
    json_data = json.dumps(overall_dictionary)
    return json_data

def processing_info():
    command = '''echo "$(iostat | head -1 | awk '{print $1}')"'''
    operating_system = return_command_output(command).decode('utf8')
    command = '''echo "$(iostat | head -1 | awk '{print $2}')"'''
    driver_version = return_command_output(command).decode('utf8')
    command = '''echo "$(iostat | head -1 | awk '{print $6}' | cut -c 2-)"'''
    total_cores = return_command_output(command).decode('utf8')
    command = '''echo "$(vmstat 1 2|tail -1|awk '{print $15}')"'''
    cpu_utilization = 100 - int(return_command_output(command).decode('utf8'))
    command = '''echo "$(vmstat --stats | grep 'total memory' | tail -1 | awk '{print $1}')"'''
    total_memory = return_command_output(command).decode('utf8')
    command = '''echo "$(vmstat --stats | grep 'used memory' | tail -1 | awk '{print $1}')"'''
    used_memory = return_command_output(command).decode('utf8')
    command = '''echo "$(df -h / | tail -1 | awk '{print $2}')"'''
    total_disk_size = return_command_output(command).decode('utf8')
    command = '''echo "$(df -h / | tail -1 | awk '{print $3}')"'''
    used_disk_size = return_command_output(command).decode('utf8')

    system_dictionary = {}
    system_dictionary["operatingSystem"] = str(operating_system)
    system_dictionary["driverVersion"] = str(driver_version)
    system_dictionary["totalCores"] = str(total_cores) 
    system_dictionary["cpuUtilization"] = str(cpu_utilization)
    system_dictionary["totalMemory"] = str(total_memory) 
    system_dictionary["usedMemory"] = str(used_memory)
    system_dictionary["totalDiskSize"] = str(total_disk_size) 
    system_dictionary["usedDiskSize"] = str(used_disk_size)

    return system_dictionary

def stream_data(data):
    headers = {'content-type':"application/json"}
    try:
        session = requests.Session()
        session.verify = False
        session.post(url='https://WAS_FQDN:443/wifimon/probes/', data=data, headers=headers, timeout=30)
    except:
        pass

def set_location_information():
    location_name = ""
    test_device_location_description = ""
    nat_network = ""
    return location_name, test_device_location_description, nat_network

def wireless_info():
    system_dictionary = processing_info()
    location_name, test_device_location_description, nat_network = set_location_information()
    iface_name = find_wlan_iface_name()
    mac = get_mac(iface_name)
    bit_rate, tx_power, link_quality, signal_level, accesspoint, essid = parse_iwconfig(iface_name)
    information = parse_iwlist(iface_name, accesspoint)
    probe_no = ""
    json_data = convert_info_to_json(accesspoint, essid, mac, bit_rate, tx_power, link_quality, signal_level, probe_no, information, location_name, test_device_location_description, nat_network, system_dictionary)
    stream_data(json_data)

if __name__ == "__main__":
    wireless_info()

The following values should be set:

  • "probe_no" (line 145) should match the number assigned to the testtools of the particular WiFiMon Hardware Probe (WHP), e.g. for the WHP assigned the number 1, the value should be "1". Assigning numbers to WHPs is possible by appropriately setting the testtool attribute included in the websites monitored by them. More information related to assigning number to WHPs is available in the WiFiMon Test Server installation guide.
  • "WAS_FQDN" (line 128) should match the FQDN of the WiFiMon Analysis Server (WAS) responsible for processing the wireless performance metrics of the WHP. The above code block assumes that the WAS uses https and port 443.
  • LInes 133 to 135 can be filled with more information regarding the location of the WHP.

Step 5: Streaming TWAMP Measurement Results to the WiFiMon Analysis Server (WAS)
Anchor
Step4
Step4

In /home/pi, you will find the Python script twping_parser.py. The contents of the script are the following:

Code Block
languagepy
firstline1
titlewireless.py
linenumberstrue
'''
Sample twping output (MIND THE NAMING OF THE LINES)

line 0: --- twping statistics from [192.168.1.1]:9706 to [192.168.1.2]:19642 ---
line 1: SID:    c0a80102e5e36a42b8a73f74cec8780e
line 2: first:  2022-03-21T23:18:58.819
line 3: last:   2022-03-21T23:19:10.456
line 4: 100 sent, 0 lost (0.000%), 0 send duplicates, 0 reflect duplicates
line 5: round-trip time min/median/max = 0.109/0.3/1.07 ms, (err=3.8 ms)
line 6: send time min/median/max = 936/936/936 ms, (err=1.9 ms)
line 7: reflect time min/median/max = -936/-936/-935 ms, (err=1.9 ms)
line 8: reflector processing time min/max = 0.00191/0.021 ms
line 9: two-way jitter = 0.1 ms (P95-P50)
line 10: send jitter = 0.1 ms (P95-P50)
line 11: reflect jitter = 0 ms (P95-P50)
line 12: send hops = 0 (consistently)
line 13:reflect hops = 0 (consistently)
'''

import subprocess
import json
import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

def return_command_output(command):
    '''
        Execute a command and return its output
    '''
    proc = subprocess.Popen(command, stdout = subprocess.PIPE, shell = True)
    (out, err) = proc.communicate()
    output = out.rstrip('\n'.encode('utf8'))
    return output

def perform_twping(twamp_server_ip):
    '''
        Perform the twping command and retrieve its output in milliseconds
    '''
    command = "twping " + str(twamp_server_ip) + " -n m -B wlan0"
    twping_results = return_command_output(command).decode('utf8')
    return twping_results

def locate_twping_data(twping_output):
    '''
        Find the line at which the important part of the twping output starts
    '''
    twping_output_parts = twping_output.split('\n')
    line_to_start = 0
    for line in twping_output_parts:
        initial_three_chars = line[0:3]
        if initial_three_chars == "---":
            break
        line_to_start += 1
    return line_to_start

# Parse lines one by one. Look at the top for the numbering of the lines
def parse_line4(line4):
    parts = line4.split(" ")
    sent, lost, send_dups, reflect_dups = parts[0], parts[2], parts[5], parts[8]
    return sent, lost, send_dups, reflect_dups

def parse_times(line):
    parts = line.split(" ")
    min_median_max = parts[4].split("/")
    minimum, median, maximum = min_median_max[0], min_median_max[1], min_median_max[2]
    err = parts[6].split("=")[1]
    return minimum, median, maximum, err

def parse_line8(line):
    parts = line.split(" ")
    time_unit = parts[-1]
    minimum = parts[-2].split("/")[0]
    maximum = parts[-2].split("/")[1]
    return minimum, maximum

def parse_jitter(line):
    parts = line.split(" ")
    value = parts[3]
    characterization = parts[5][1:-1]
    return value, characterization

def parse_hops(line):
    parts = line.split(" ")
    value = parts[3]
    characterization = parts[4][1:-1]
    return value, characterization

def form_json(probe_number, twamp_server, sent, lost, send_dups, reflect_dups, 
        min_rtt, median_rtt, max_rtt, err_rtt, min_send, median_send, max_send, 
        err_send, min_reflect, median_reflect, max_reflect, err_reflect, 
        min_reflector_processing_time, max_reflector_processing_time,
        two_way_jitter_value, two_way_jitter_char, send_jitter_value, send_jitter_char,
        reflect_jitter_value, reflect_jitter_char, send_hops_value, send_hops_char,
        reflect_hops_value, reflect_hops_char):
    '''
        Create a json object with the parsed values. Values are first stored in a dictionary.
    '''
    overall_dictionary = {}
    overall_dictionary["probeNumber"] = probe_number
    overall_dictionary["twampServer"] = twamp_server
    overall_dictionary["sent"] = sent
    overall_dictionary["lost"] = lost
    overall_dictionary["sendDups"] = send_dups
    overall_dictionary["reflectDups"] = reflect_dups
    overall_dictionary["minRtt"] = min_rtt
    overall_dictionary["medianRtt"] = median_rtt
    overall_dictionary["maxRtt"] = max_rtt
    overall_dictionary["errRtt"] = err_rtt
    overall_dictionary["minSend"] = min_send
    overall_dictionary["medianSend"] = median_send
    overall_dictionary["maxSend"] = max_send
    overall_dictionary["errSend"] = err_send
    overall_dictionary["minReflect"] = min_reflect
    overall_dictionary["medianReflect"] = median_reflect
    overall_dictionary["maxReflect"] = max_reflect
    overall_dictionary["errReflect"] = err_reflect
    overall_dictionary["minReflectorProcessingTime"] = min_reflector_processing_time
    overall_dictionary["maxReflectorProcessingTime"] = max_reflector_processing_time
    overall_dictionary["twoWayJitterValue"] = two_way_jitter_value
    overall_dictionary["twoWayJitterChar"] = two_way_jitter_char
    overall_dictionary["sendJitterValue"] = send_jitter_value
    overall_dictionary["sendJitterChar"] = send_jitter_char
    overall_dictionary["reflectJitterValue"] = reflect_jitter_value
    overall_dictionary["reflectJitterChar"] = reflect_jitter_char
    overall_dictionary["sendHopsValue"] = send_hops_value
    overall_dictionary["sendHopsChar"] = send_hops_char
    overall_dictionary["reflectHopsValue"] = reflect_hops_value
    overall_dictionary["reflectHopsChar"] = reflect_hops_char
    json_data = json.dumps(overall_dictionary)
    return json_data

def parse_twping(twping_output, line_to_start, probe_number):
    '''
        Parse twping output line by line
    '''
    twping_output_parts = twping_output.split('\n')
    sent, lost, send_dups, reflect_dups = parse_line4(twping_output_parts[line_to_start + 4])
    min_rtt, median_rtt, max_rtt, err_rtt = parse_times(twping_output_parts[line_to_start + 5])
    min_send, median_send, max_send, err_send = parse_times(twping_output_parts[line_to_start + 6])
    min_reflect, median_reflect, max_reflect, err_reflect = parse_times(twping_output_parts[line_to_start + 7])
    min_reflector_processing_time, max_reflector_processing_time = parse_line8(twping_output_parts[line_to_start +8])
    two_way_jitter_value, two_way_jitter_char = parse_jitter(twping_output_parts[line_to_start + 9])
    send_jitter_value, send_jitter_char = parse_jitter(twping_output_parts[line_to_start + 10])
    reflect_jitter_value, reflect_jitter_char = parse_jitter(twping_output_parts[line_to_start + 11])
    send_hops_value, send_hops_char = parse_hops(twping_output_parts[line_to_start + 12])
    reflect_hops_value, reflect_hops_char = parse_hops(twping_output_parts[line_to_start + 13])
    json_data = form_json(probe_number, twamp_server, sent, lost, send_dups, reflect_dups, 
            min_rtt, median_rtt, max_rtt, err_rtt, min_send, median_send, max_send, err_send, 
            min_reflect, median_reflect, max_reflect, err_reflect, min_reflector_processing_time,
            max_reflector_processing_time, two_way_jitter_value, two_way_jitter_char, 
            send_jitter_value, send_jitter_char, reflect_jitter_value, reflect_jitter_char, 
            send_hops_value, send_hops_char, reflect_hops_value, reflect_hops_char)
    return json_data

def stream_data(json_data):
    '''
        Stream JSON data to the WiFiMon Analysis Server
        Set the FQDN of the WiFiMon Analysis Server
    '''
    headers = {'content-type' : "application/json"}
    try:
        session = requests.Session()
        session.verify = False
        session.post(url = 'https://WAS_FQDN:443/wifimon/probestwamp/', data = json_data, headers = headers, timeout = 30)
    except:
        pass

def set_location_information():
    locationreturn None

if __name__ == ""
    test_device_location_description = ""__main__":
    nat_network# =Define ""
the    return location_name, test_device_location_description, nat_network

def wireless_info():
    location_name, test_device_location_description, nat_network = set_location_information()
    iface_name = find_wlan_iface_name()
    mac = get_mac(iface_name)
    bit_rate, tx_power, link_quality, signal_level, accesspoint, essid = parse_iwconfig(iface_name)
    information = parse_iwlist(iface_name, accesspoint)
    probe_no = ""
    json_data = convert_info_to_json(accesspoint, essid, mac, bit_rate, tx_power, link_quality, signal_level, probe_no, information, location_name, test_device_location_description, nat_network)
    stream_data(json_data)

if  == "":
    wireless_info(number of the WiFiMon Hardware Probe
    PROBE_NO = "PROBE_NUMBER"
    # Define the FQDN of the TWAMP Server
    twamp_server = "TWAMP_SERVER_FQDN"
    # Perform twping against the TWAMP Server
    twping_results = perform_twping(twamp_server)
    # Parse twping results
    line_to_start = locate_twping_data(twping_results)
    json_data = parse_twping(twping_results, line_to_start, PROBE_NO)
    # Stream data to the WiFiMon Analysis Server
    stream_data(json_data)

The following values should be set:

  • "probePROBE_noNO" (line 109171) should match the number assigned to the testtools of the particular WiFiMon Hardware Probe (WHP), e.g. for the WHP assigned the number 1,  the the value should be "1". Assigning numbers to WHPs is possible by appropriately setting the testtool attribute included in the websites monitored by them. More information related to assigning number to WHPs is available in the WiFiMon Test Server installation guide.
  • "WAS_FQDN" (line 93164) should match the FQDN of the WiFiMon Analysis Server (WAS) responsible for processing the wireless performance metrics TWAMP measurement results of the WHP. The above code block assumes that the WAS uses https and port 443.
  • LInes 98 to 100 can "TWAMP_SERVER_FQDN" (line 173): Should be filled with more information regarding the location FQDN of the WHP.

...

  • TWAMP Server.

Security Issues

We suggest that you take additional efforts to safeguard the security of your probes:

...