Page History

Quick Reference

For those who definitely know what they're doing and just need a reminder of the commands:

ssh -l dante.<surname> prod-cacti01-fra-de.geant.net
chmod 777 to the /opt/cacti/rra/<name_of_rrd>.rrd <-- Richard has permissions to do this step, Ops does not!
cd /tmp
rrdtool dump /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd > /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml
vi /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml
sudo -u cactiuser mv /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd.bak
sudo -u cactiuser rrdtool restore /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd
sudo -u cactiuser scp -i /home/cactiuser/.ssh/id_dsa /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd cactiuser@prod-cacti02-vie-at.geant.net:/opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd

Detailed Instructions

Locate the graph with the Spike:

...

In this case the rrd file name is /opt/cacti-0.8.7e/rra/mx1_lon_uk_traffic_in_20683.rrd. In the remainder of this description we will use this rrd file. In your case you will need to change the name appropriately. You will also need to remove the Cacti version number from the path, as the real path is /opt/cacti/rra.

Click on the Magnifying Glass icon to the right of the graph and select the time period you wish to zoom into.

...

Continue 'zooming' until you can determine the date and time range of the Spikes. In the above case one spike occurred on 19/01/2016 and the other on 26/01/2016

Using the Epoch & Unix Timestamp Conversion Tool found at http://www.epochconverter.com/ convert the date/times into their UTC equivalent as Cacti stored its date and time in UTC format.

In this case the UTC ranges are 1453161601 - 1453247999 and 1453766401 - 1453852799. We can also see that the data values are in the range of 3x10e12 (3.0T) and 3.5x10e12 (3.5T)

Log into prod-cacti01-fra-de.geant.net. Username will be of the format dante.<surname> and the password will be the same as in Crowd and used forLog into prod-cacti01-fra-de.geant.net issues.geant.net.

Change to the /tmp directory and export the data to an XML file format.

Check that the chmod of the rrd file is set to 7777 prior to attempting to remove a spike.

rrdtool dump /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd > /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml

Edit the /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml and find the offending data. Change the values to a more meaningful order of magnitude. Save your changes.. As we know the data is in the order of 10e11, we can search for the following string e+11. We can see the timestamp is in the right range.

NOTE: The values in the RRD file are in bytes per second. You will need to multiple these values by 8 to obtain the bits per second. The graphs on the website and CSV files that can be downloaded from the website all show the traffic levels in bits per second.

Inbound traffic:

<row><v> 3.5642652369e+07 </v><v> 1.1136736365e+06 </v></row>
 <row><v> 9.0115656678e+07 </v><v> 3.1514951986e+06 </v></row>
 <row><v> 3.8543163654e+11 </v><v> 9.2684560857e+09 </v></row>
 <row><v> 8.0043154538e+07 </v><v> 2.7647348898e+07 </v></row>
 <row><v> 1.0178635224e+08 </v><v> 2.1954640988e+07 </v></row>
 <row><v> 6.5086071137e+07 </v><v> 2.7845010247e+07 </v></row>

<row><v> 3.0217637424e+07 </v><v> 2.4722224375e+07 </v></row>
 <row><v> 7.9918286563e+07 </v><v> 2.1599139048e+07 </v></row>
 <row><v> 4.4333612638e+11 </v><v> 2.0011783987e+10 </v></row>
 <row><v> 7.5003371999e+07 </v><v> 1.8443001057e+06 </v></row>
 <row><v> 7.2175201113e+07 </v><v> 4.0701536470e+05 </v></row>
 <row><v> 6.3959119353e+07 </v><v> 1.9703085506e+05 </v></row>

You can observe that around the spike, the data was in the order of 10e7, so a more meaningful data value would be to do a search and replace for e+11 and replace it with e+07.

Outbound traffic:

<row><v> 9.0115656678e+07 </v><v> 3.1514951986e+06 </v></row>
 <row><v> 3.8543163654e+07 </v><v> 9.2684560857e+09 </v></row>
 <row><v> 8.0043154538e+07 </v><v> 2.7647348898e+07 </v></row>

Take great care using global search and replace, do a find first to see how many data items are in this order. In the case of 10e9, there are a few so you will have to be more specific in the search and replace e.g find 9.2684560857e+09 and replace with 9.2684560857e+06

<row><v> 3.0217637424e+07 </v><v> 2.4722224375e+07 </v></row>
 <row><v> 7.9918286563e+07 </v><v> 2.1599139048e+07 </v></row>
 <row><v> 4.4333612638e+07 </v><v> 2.0011783987e+10 </v></row>
 <row><v> 7.5003371999e+07 </v><v> 1.8443001057e+06 </v></row>
 <row><v> 7.2175201113e+07 </v><v> 4.0701536470e+05 </v></row>

You can observe that around the spike, the data was in the order of 10e7, so a more meaningful data value would be to do a search and replace for e+10 and replace it with e+07.

Make the necessary changes in the text editor, then save your changes. (You may find it easier to WinSCP this file to your local machine for editing in a different text editor if you are not famiiar with UNIX text editors such as vi).

Create a backup of the file you wish to change. . Please note we are using mv and not cp because we will be unable to restore a rrd file if it exists already on the file system.

sudo -u cactiuser mv /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd.bak

Recreate the RRD file using the data from the exported XML file.

sudo -u cactiuser rrdtool restore /tmp/mx1_lon_uk_traffic_in_20683.rrd.xml /opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd

We now need to sync this change with the other production instance of Cacti on prod-cacti02-vie-at.geant.net

sudo -u cactiuser scp -i /home/cactiuser/.ssh/id_dsa cactiuser@prod-cacti02-vie-at.geant.net:/opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrd .rrd cactiuser@prod-cacti02-vie-at.geant.net:/opt/cacti/rra/mx1_lon_uk_traffic_in_20683.rrdrrd

The graph should now look a lot better:

Image Added

Page tree

Versions Compared

Old Version 2

New Version Current

Key

Quick Reference

Detailed Instructions