Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Time (CET)
18 Jun, 23:16Issue Reported by OC
19 Jun, 07:35

Picked up by Michael H the following morning

19 Jun, 08:30

Fixed by turning off SSL temporarily to restore the service. Initial investigation revealed certificate has expired but later turned out that wasn't the case.

19 Jun, 10:30

Further investigations were carried out to avoid such failures in future

19 Jun, 16:08

The actual cause identified for the failure - due to IT patching certificates were automatically changed.

20 Jun, 09:30

Proposal was discussed between IT and SWD to avoid such failures in future.

20 Jun, 11:30

Part one of Nagios check in the proposed solution implemented



Total downtime: 09:14 hours.

...

1) Any PKI patches on domain controller should go via change control and . The changes should be communicated to DevOps by IT so that preventive measures could be taken in timely manner.

...