Problem
New alarms are not appearing in the gui
Possible Cause
Traps not being processed by RabbitMQ
Analysis
- Open one or more of the following RabbitMQ management consoles. (Credentials are in the "GÉANT Dashboard v3" LastPass folder)
- Scroll down to the "Nodes" section
- There should be 3 rows in the table and all status icons should be green. The expected node hostnames are:
- prod-noc-alarms01.geant.org
- prod-noc-alarms02.geant.org
- prod-dashboard-storage01.geant.org
Solution
- If one of the 3 nodes is failing or missing from the list, log into the failing server and restart the RabbitMQ service:
sudo systemctl restart rabbitmq-server
- After a minute or two the management consoles should show the cluster is restored.
Possible Cause
Collectors have stopped working
Analysis
- Open https://net-alarms-monitoring.geant.org/d/hESYQotZz/correlation-services?orgId=1
- Scroll down to the "Collectors" panel
- Check that the graph shows a nonzero rate of traps being processes
Solution
- On each of the following servers:
- net-alarms01.geant.org
- net-alarms02.geant.org
- Log in and execute the following command:
sudo systemctl restart trap_collector
Possible Cause
Correlators have stopped working
Analysis
- Open https://net-alarms-monitoring.geant.org/d/hESYQotZz/correlation-services?orgId=1
- Scroll down to the "Collectors" panel
- Check that the graph shows one of the collector processes processing a non-zero rate of traps.
- note that it is normal for only one of the collectors to be processing traps, the other line should remain at zero.
Solution
- On each of the following servers:
- Log in and execute the following command:
sudo systemctl restart trap_correlator
Related articles