How To Remove Spikes From Cacti Graphs

Quick Reference

For those who definitely know what they're doing and just need a reminder of the commands:

ssh -l dante.<surname> prod-cacti01-fra-de.geant.net
chmod 777 to the /opt/cacti/rra/<name_of_rrd>.rrd <-- Richard has permissions to do this step, Ops does not!
cd /tmp
rrdtool dump /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd > /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml
vi /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml
sudo -u cactiuser mv /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd.bak
sudo -u cactiuser rrdtool restore /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd
sudo -u cactiuser scp -i /home/cactiuser/.ssh/id_dsa /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd cactiuser@prod-cacti02-vie-at.geant.net:/opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd

Detailed Instructions

Locate the graph with the Spike:

Click on the Spanner icon to the right of the graph to determine the name of the Round Robin Database (RRD) file that requires changing.

In this case the rrd file name is /opt/cacti-0.8.7e/rra/mx1_lon_uk_traffic_in_20683.rrd. In the remainder of this description we will use this rrd file. In your case you will need to change the name appropriately. You will also need to remove the Cacti version number from the path, as the real path is /opt/cacti/rra.

Click on the Magnifying Glass icon to the right of the graph and select the time period you wish to zoom into.

Continue 'zooming' until you can determine the date and time range of the Spikes. In the above case one spike occurred on 19/01/2016 and the other on 26/01/2016

Log into prod-cacti01-fra-de.geant.net. Username will be of the format dante.<surname> and the password will be the same as in Crowd and used for issues.geant.net.

Change to the /tmp directory and export the data to an XML file format.

Check that the chmod of the rrd file is set to 7777 prior to attempting to remove a spike.

rrdtool dump /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd > /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml

Edit the /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml and find the offending data. As we know the data is in the order of 10e11, we can search for the following string e+11. We can see the timestamp is in the right range.

NOTE: The values in the RRD file are in bytes per second. You will need to multiple these values by 8 to obtain the bits per second. The graphs on the website and CSV files that can be downloaded from the website all show the traffic levels in bits per second.

Inbound traffic:

<row><v> 3.5642652369e+07 </v><v> 1.1136736365e+06 </v></row>
 <row><v> 9.0115656678e+07 </v><v> 3.1514951986e+06 </v></row>
 <row><v> 3.8543163654e+11 </v><v> 9.2684560857e+09 </v></row>
 <row><v> 8.0043154538e+07 </v><v> 2.7647348898e+07 </v></row>
 <row><v> 1.0178635224e+08 </v><v> 2.1954640988e+07 </v></row>
 <row><v> 6.5086071137e+07 </v><v> 2.7845010247e+07 </v></row>

<row><v> 3.0217637424e+07 </v><v> 2.4722224375e+07 </v></row>
 <row><v> 7.9918286563e+07 </v><v> 2.1599139048e+07 </v></row>
 <row><v> 4.4333612638e+11 </v><v> 2.0011783987e+10 </v></row>
 <row><v> 7.5003371999e+07 </v><v> 1.8443001057e+06 </v></row>
 <row><v> 7.2175201113e+07 </v><v> 4.0701536470e+05 </v></row>
 <row><v> 6.3959119353e+07 </v><v> 1.9703085506e+05 </v></row>

You can observe that around the spike, the data was in the order of 10e7, so a more meaningful data value would be to do a search and replace for e+11 and replace it with e+07.

Outbound traffic:

<row><v> 9.0115656678e+07 </v><v> 3.1514951986e+06 </v></row>
 <row><v> 3.8543163654e+07 </v><v> 9.2684560857e+09 </v></row>
 <row><v> 8.0043154538e+07 </v><v> 2.7647348898e+07 </v></row>

Take great care using global search and replace, do a find first to see how many data items are in this order. In the case of 10e9, there are a few so you will have to be more specific in the search and replace e.g find 9.2684560857e+09 and replace with 9.2684560857e+06

<row><v> 3.0217637424e+07 </v><v> 2.4722224375e+07 </v></row>
 <row><v> 7.9918286563e+07 </v><v> 2.1599139048e+07 </v></row>
 <row><v> 4.4333612638e+07 </v><v> 2.0011783987e+10 </v></row>
 <row><v> 7.5003371999e+07 </v><v> 1.8443001057e+06 </v></row>
 <row><v> 7.2175201113e+07 </v><v> 4.0701536470e+05 </v></row>

You can observe that around the spike, the data was in the order of 10e7, so a more meaningful data value would be to do a search and replace for e+10 and replace it with e+07.

Make the necessary changes in the text editor, then save your changes. (You may find it easier to WinSCP this file to your local machine for editing in a different text editor if you are not famiiar with UNIX text editors such as vi).

Create a backup of the file you wish to change. Please note we are using mv and not cp because we will be unable to restore a rrd file if it exists already on the file system.

sudo -u cactiuser mv /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd.bak

Recreate the RRD file using the data from the exported XML file.

sudo -u cactiuser rrdtool restore /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd

We now need to sync this change with the other production instance of Cacti on prod-cacti02-vie-at.geant.net

sudo -u cactiuser scp -i /home/cactiuser/.ssh/id_dsa /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd cactiuser@prod-cacti02-vie-at.geant.net:/opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd

The graph should now look a lot better:

Page tree

How To Remove Spikes From Cacti Graphs

Quick Reference

Detailed Instructions