Page History

The configuration files are stored in GitLab at the location: https://git.geant.net/gitlab/puppet-apps/dashboard/tree/test. You have may have to ask DevOps to grant you access to the dashboard project

...

The physical file on the server is: /usr/lib64/nagios/plugins/dbupdates/conf/routers_community.conf (Should you wish to be naughty and update the file directly on the server!!!)

*** Because the Operations team has commissioned a new router, we want to Dashboard to be able to raise critical alarms should a fault arise. In order for this to happen, we need to run some of the PHP and BASH Shell scripts detailed in the CRONTAB.

Log into prod-newdboard01.geant.net using your dante.surname credentials.

...

We are now going to manually run the commands highlighted in bold text:

*** Some of the scripts use a configuration files located in file called /usr/lib64/nagios/plugins/dbupdates/conf/config.inc

cd /usr/lib64/nagios/plugins/dbupdates/scripts/

...

./populate_router_details_table.php - This inserts records in the alarms.router_details table (This can take some time to complete)

./get-bgp-v4-peers-on-router.sh -  It trys to PING each router listed in the /usr/lib64/nagios/plugins/dbupdates/conf/routers_community.conf file, if it gets a response, it then connects to each router using ssh and public keys (/usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_dsa and /usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_known_hosts). It generates a file called /var/lib/nms-dashboard/v4-<Router_Name>.txttxt  (This can take about 10 minutes to complete)

./get-bgp-v6-peers-on-router.sh -  It trys to PING each router listed in the /usr/lib64/nagios/plugins/dbupdates/conf/routers_community.conf file, if it gets a response, it then connects to each router using ssh and public keys (/usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_dsa and  /usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_known_hosts). It generates a file called /var/lib/nms-dashboard/v6-<Router_Name>.txttxt (This can take about 10 minutes to complete)

./populate_router_ix_public_peer_table.php -  This parses the .txt files generated by get-bgp-v4-peers-on-router.sh and get-bgp-v6-peers-on-router.sh, located in the the /var/lib/nms-dashboard directory. For each .txt file found, it updates/deletes records from the alarms.router_ix_public_peers table

*** WARNING - The scripts run the equivalent ssh command:

ssh -n -i "/usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_dsa" -oPasswordAuthentication=no -oStrictHostKeyChecking=no -oPreferredAuthentications=publickey -oConnectionAttempts=1 -oConnectTimeout=10 -oUserKnownHostsFile=/usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_known_hosts Monit0r@mx1.dub.ie.geant.net

The scripts may print out an error/warning such as:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The RSA host key for mx1.dub.ie.geant.net has changed,
and the key for the corresponding IP address 62.40.96.3
has a different value. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
Offending key for IP in /usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_known_hosts:135

To fix the above, we need to delete the line that references the router within the /usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_known_hosts, to do this easily do the following:

Change to root by running: sudo su -

cd /usr/lib64/nagios/plugins/dbupdates/.sshkeys/

chmod 666 monitor_known_hosts

Now edit the file and remove all references to the router (both IPv4 and IPv6 references), this could be by router name and by ip address

chmod 600 monitor_known_hosts

Now rerun the command: ssh -n -i "/usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_dsa" -oPasswordAuthentication=no -oStrictHostKeyChecking=no -oPreferredAuthentications=publickey -oConnectionAttempts=1 -oConnectTimeout=10 -oUserKnownHostsFile=/usr/lib64/nagios/plugins/dbupdates/.sshkeys/monitor_known_hosts Monit0r@mx1.dub.ie.geant.net

You should see the following:

Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added the RSA host key for IP address '62.40.96.3' to the list of known hosts.
----------------------------------------------------------------

This is mx1.dub.ie.geant.net, a GEANT router in Dublin, Ireland
Warning: Unauthorized access to this equipment is strictly forbidden and will lead to prosecution

-------------------------------------------------------------
{master}
Monit0r@mx1.dub.ie.re0>

Once you have fixed the monitor_known_hosts file, you we need to rerun the get-bgp-v4-peers-on-router.sh, get-bgp-v6-peers-on-router.sh and populate_router_ix_public_peer_table.php scripts.

The router's details also need to be added manually to the juniper_servers table in the alarms database.

1. Log into the alarms database using the newdashboardBizuser
2. execute the follow statement INSERT INTO juniper_servers VALUES ('<ip_address>', '<router project>');