UPDATE ......From Tuesday 8 April 2025 we have changed the way that Single Sign-on works on this wiki. Please see here for more information:
Update
Log into http://cacti.geant.net/cacti/
Locate the graph with the Spike:
Click on the Spanner icon to the right of the graph to determine the name of the Round Robin Database (RRD) file that requires changing.
In this case the rrd file name is /opt/cacti-0.8.7e/rra/mx1_lon_uk_traffic_in_20683.rrd. In the remainder of this description we will use this rrd file. In your case you will need to change the name appropriately. You will also need to remove the Cacti version number from the path, as the real path is /opt/cacti/rra.
Click on the Magnifying Glass icon to the right of the graph and select the time period you wish to zoom into.
Continue 'zooming' until you can determine the date and time range of the Spikes. In the above case one spike occurred on 19/01/2016 and the other on 26/01/2016
Using the Epoch & Unix Timestamp Conversion Tool found at http://www.epochconverter.com/ convert the date/times into their UTC equivalent as Cacti stored its date and time in UTC format.
In this case the UTC ranges are 1453161601 - 1453247999 and 1453766401 - 1453852799. We can also see that the data values on the graph are in the range of 3x10e12 (3.0Tbits) and 3.5x10e12 (3.5Tbits). The RRD Files stores the information in Bytes so we have to divide by 8 to get the magnitude of bytes. (The are 8 bits in a byte). In this case we are looking for data values in the RRD file around 3.75x10e11 and 4.375x10e11. With respect to the outbound traffic the data values on the graph are 7x10e7 (70Gbits) and 1.5x10e11 (150Gbits), so we should look for data in the order of 8.75x10e9 and 1.875x10e10
Log into prod-cacti01-fra-de.geant.net
Change to the /tmp directory and export the data to an XML file format.
rrdtool dump /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd > /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml
Edit the /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml and find the offending data. As we know the data is in the order of 10e11, we can search for the following string e+11. We can see the timestamp is in the right range.
Inbound traffic:
<!-- 2016-01-18 00:00:00 UTC / 1453075200 --> <row><v> 3.5642652369e+07 </v><v> 1.1136736365e+06 </v></row>
<!-- 2016-01-19 00:00:00 UTC / 1453161600 --> <row><v> 9.0115656678e+07 </v><v> 3.1514951986e+06 </v></row>
<!-- 2016-01-20 00:00:00 UTC / 1453248000 --> <row><v> 3.8543163654e+11 </v><v> 9.2684560857e+09 </v></row>
<!-- 2016-01-21 00:00:00 UTC / 1453334400 --> <row><v> 8.0043154538e+07 </v><v> 2.7647348898e+07 </v></row>
<!-- 2016-01-22 00:00:00 UTC / 1453420800 --> <row><v> 1.0178635224e+08 </v><v> 2.1954640988e+07 </v></row>
<!-- 2016-01-23 00:00:00 UTC / 1453507200 --> <row><v> 6.5086071137e+07 </v><v> 2.7845010247e+07 </v></row>
<!-- 2016-01-25 00:00:00 UTC / 1453680000 --> <row><v> 3.0217637424e+07 </v><v> 2.4722224375e+07 </v></row>
<!-- 2016-01-26 00:00:00 UTC / 1453766400 --> <row><v> 7.9918286563e+07 </v><v> 2.1599139048e+07 </v></row>
<!-- 2016-01-27 00:00:00 UTC / 1453852800 --> <row><v> 4.4333612638e+11 </v><v> 2.0011783987e+10 </v></row>
<!-- 2016-01-28 00:00:00 UTC / 1453939200 --> <row><v> 7.5003371999e+07 </v><v> 1.8443001057e+06 </v></row>
<!-- 2016-01-29 00:00:00 UTC / 1454025600 --> <row><v> 7.2175201113e+07 </v><v> 4.0701536470e+05 </v></row>
<!-- 2016-01-30 00:00:00 UTC / 1454112000 --> <row><v> 6.3959119353e+07 </v><v> 1.9703085506e+05 </v></row>
You can observe that around the spike, the data was in the order of 10e7, so a more meaningful data value would be to do a search and replace for e+11 and replace it with e+07.
Outbound traffic:
<!-- 2016-01-19 00:00:00 UTC / 1453161600 --> <row><v> 9.0115656678e+07 </v><v> 3.1514951986e+06 </v></row>
<!-- 2016-01-20 00:00:00 UTC / 1453248000 --> <row><v> 3.8543163654e+07 </v><v> 9.2684560857e+09 </v></row>
<!-- 2016-01-21 00:00:00 UTC / 1453334400 --> <row><v> 8.0043154538e+07 </v><v> 2.7647348898e+07 </v></row>
Take great care using global search and replace, do a find first to see how many data items are in this order. In the case of 10e9, there are a few so you will have to be more specific in the search and replace e.g find 9.2684560857e+09 and replace with 9.2684560857e+06
<!-- 2016-01-25 00:00:00 UTC / 1453680000 --> <row><v> 3.0217637424e+07 </v><v> 2.4722224375e+07 </v></row>
<!-- 2016-01-26 00:00:00 UTC / 1453766400 --> <row><v> 7.9918286563e+07 </v><v> 2.1599139048e+07 </v></row>
<!-- 2016-01-27 00:00:00 UTC / 1453852800 --> <row><v> 4.4333612638e+07 </v><v> 2.0011783987e+10 </v></row>
<!-- 2016-01-28 00:00:00 UTC / 1453939200 --> <row><v> 7.5003371999e+07 </v><v> 1.8443001057e+06 </v></row>
<!-- 2016-01-29 00:00:00 UTC / 1454025600 --> <row><v> 7.2175201113e+07 </v><v> 4.0701536470e+05 </v></row>
You can observe that around the spike, the data was in the order of 10e7, so a more meaningful data value would be to do a search and replace for e+10 and replace it with e+07.
Make the necessary changes in the text editor, then save your changes. (You may find it easier to WinSCP this file to your local machine for editing in a different text editor)
Create a backup of the file you wish to change. Please note we are using mv and not cp because we will be unable to restore a rrd file if it exists already on the file system.
sudo -u cactiuser mv /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd.bak
Recreate the RRD file using the data from the exported XML file.
sudo -u cactiuser rrdtool restore /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd
We now need to sync this change with the other production instance of Cacti on prod-cacti02-vie-at.geant.net
sudo -u cactiuser scp -i /home/cactiuser/.ssh/id_dsa /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd cactiuser@prod-cacti02-vie-at.geant.net:/opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd
The graph should now look a lot better:
