...
Timeline
all times here in CET
1415:06 - foreman triggered vm remove action.
15:35 - Dick Visser contacted us about problems with wordpress and
filesender because he got monitoring alert that sites are not accessible
15:41 - Konstantin Lepikhov confirmed that hosts are missing in vmware
15:44 - Konstantin Lepikhov identified the issue and started investigation on vmware cluster
16:43 - Konstantin Lepikhov contacted Qaiser in Slack to confirm backup existence
16:45 - DevOps confirmed that there are no backups or extra copies on
vmware storage
17:00 - Konstantin Lepikhov called Qaiser Ahmed in Slack, no response.
...
17:00 - Dick Visser confirmed that he has backups on server at
Amsterdam university (those are daily backups taken directly by VMs
itself).
...