Incident Description

BRIAN bitrate traffic was not computed or saved for approximately 21 hours.


The reason for degradation:


The impact of this service degradation was:


Incident severity:  Intermittent service outage

Data loss: 

Total duration of incident: 21 hours/On going (as of  22:22 UTC)


Timeline

All times are in UTC

DateTime (UTC)Description

 

12:52:37

the first evidence of this incident appeared in the logs of prod-poller-processor.geant.org


May 30 12:52:37 prod-poller-processor kapacitord[124994]: ts=2022-05-30T12:52:37.802Z lvl=error msg="failed to write points to InfluxDB" service=kapacitor task_master=main task=remove_spikes_gwsd_rates node=influxdb_out3 err=timeout

May 30 12:52:38 prod-poller-processor kapacitord[124994]: ts=2022-05-30T12:52:38.069Z lvl=error msg="encountered error" service=kapacitor task_master=main task=remove_spikes_interface_rates node=remove_spikes2 err="keepalive timedout, last keepalive received was: 2022-05-30 12:52:28.069298439 +0000 UTC"


 

12:53First user query about EMS login problem (Slack #it)

 

08:12

System was stopped

 

08:26:55

System rebooting and starting up

 

08:26:55

haproxy failed to start because it couldn't resolve prod-inventory-provider0x.geant.org


May 31 08:26:55 prod-poller-processor haproxy[976]: [ALERT] 150/082655 (976) : parsing [/etc/haproxy/haproxy.cfg:30] : 'server prod-inventory-provider01.geant.org' : could not resolve address 'prod-inventory-provider01.geant.org'.

May 31 08:26:55 prod-poller-processor haproxy[976]: [ALERT] 150/082655 (976) : parsing [/etc/haproxy/haproxy.cfg:31] : 'server prod-inventory-provider02.geant.org' : could not resolve address 'prod-inventory-provider02.geant.org'.


 

20:50

Kapacitor tasks failed to run because the haproxy service wasn't running, for example:

May 31 08:27:07 prod-poller-processor kapacitord[839]: ts=2022-05-31T08:27:07.962Z lvl=info msg="UDF log" service=kapacitor task_master=main task=service_enrichment node=inventory_enrichment2 text="urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /poller/interfaces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f749f4a2978>: Failed to establish a new connection: [Errno 111] Connection refused',))"






Proposed Solution