...
A missing DNS record for dndrdc01.win.dante.org.uk caused the Haproxy service to fail to restart (the . The server dndrdc01 was decommissioned on 22 May 2020, but the record was cleaned up recently. We still don't know when this record was expunged (we don't have access to Windows DNS server), and we know that today morning even uat-haproxy went down for the same reason.
A change request was not raised because this is an unattended job, which is run nightly, every day and I have only run in advance the job that would have been triggered in the night. The first certificate that was going to expire would have caused the same issue. We could not imagine that a DNS record was deleted the same day.
Incident severity: CRITICAL
...