Background

Every Four Hours (which is configurable) or when a user presses the Refresh Map button a message is created on the Dashboard Enterprise Service Bus (ESB) in the prodrepopulate.map.queue

The OpsDB Copier process is a runnable JAR file that takes a MySQL dump of the OpsDB data and stores it in /home/local/GEANT/dante.sanigar/dumps/opsdb.sql this file is tarred up into  /home/local/GEANT/dante.sanigar/dumps.tar.gz

The file is rsynced across to the three instances of the Correlator machines:  test-dboard01-corr.geant.net, uat-dboard01-corr.geant.net and prod-dboard01-corr.geant.net to /tmp/received/dumps.tar.gz 

The process writes to a log file /home/local/GEANT/dante.sanigar/copier.log. Changes to the configuration can be made by editing the  /home/local/GEANT/dante.sanigar/config/application.properties 

When the OpsDB Copier finishes, it sends a message to the ESB prodreload.map.queue.

When a message appears in the prodreload.map.queue, the ESB Consumer on the Correlator machine picks up the message and starts to load the data it has obtained from the Operations Database (OpsDB) into the Dashboard Network Map. The file /tmp/received/dumps.tar.gz is untarred to /tmp/received/opsdb.sql and then processed

The Consumer on the Correlator machine is a Tomcat7 webapp called /var/lib/apache-tomcat7/webapps/populate-web

 Apart from the Tomcat Logs located on /var/log/apache-tomcat7, three log files are created in the /var/log/snmptraps/new-dashboard/Populator/ directory: These are allLog.txt, errorLog.txt and warnLog.txt

The configuration files are stored in /var/lib/apache-tomcat7/webapps/populate-web/WEB-INF/classes. 

Check ESB

Check the prodrepopulate.map.queue on http://prod-dboard01-esb.geant.net:8161/admin/browse.jsp?JMSDestination=prodrepopulate.map.queue for messages being processed.

Also check the the consumer is running: http://prod-dboard01-esb.geant.net:8161/admin/queueConsumers.jsp?JMSDestination=prodrepopulate.map.queue 

Check the prodreload.map.queue on http://prod-dboard01-esb.geant.net:8161/admin/browse.jsp?JMSDestination=prodreload.map.queue for messages being processed.

Also check the the consumer is running: http://prod-dboard01-esb.geant.net:8161/admin/queueConsumers.jsp?JMSDestination=prodreload.map.queue

If the ESB pages cannot be displayed, then you may need to restart the ActiveMQ broker service on the ESB. Login to the prod-dboard01-esb.geant.net and run as sudo service activemq restart

Check prod-opsdb01.geant.net

First things to check is whether the OpsDB Copier Process is running on prod-opsdb01.geant.net.

The configuration file for the process is stored in /home/local/GEANT/dante.sanigar/config/application.properties

Navigate to /home/local/GEANT/dante.sanigar

Check that the process is running: pgrep java

Check the last time the file opsdb.sql was created: ls -lart dumps

Check the contents of the copier.log to see for any obvious errors: tail -50 copier.log

If the process is not running, it can be started: java -jar opsdbCopier-1.0-SNAPSHOT.jar > copier.log &

Check prod-dboard01-corr.geant.net

Next check that everything is OK on the Correlator box prod-dboard01-corr.geant.net

Login into prod-dboard01-corr.geant.net

Check the timestamp of the file /tmp/received : ls -lart received

Check the Map Repopulator log files: tail -150 /var/log/snmptraps/new-dashboard/Populator/allLog.txt

Check that Tomcat7 is running: service apache-tomcat7 status