Incident Description

Users of EMS could not logon to EMS. They were presented with the login screen, which took them to the IDP selection page (as per normal). After successful authentication on the IDP, they where redirected to EMS. However, instead of being logged in on EMS, they where logged out.

The reason for degradation:

EMS/Indico stores user sessions in redis
prod-events01.geant.org and prod-events02.geant.org could not resolve the hostname for the redis server (master.production-events-redis.service.ha.geant.net)
With the connection to redis lost, Indico could not create or manage user sessions

The impact of this service degradation was:

Users could not manage their events, for example:
- Editing the event
- Accessing registation lists
- Sending out reminder emails

Incident severity: Partial service degradation

Data loss:

Total duration of incident: 15 hours

Timeline

All times are in UTC

Date	Time	Description
19 Apr 2022	21:55:53	First error in indico.log of redis being unavailable: ConnectionError: Error -2 connecting to master.production-events-redis.service.ha.geant.net:6379. Name or service not known.
20 Apr 2022	10:42	First user query about EMS login problem (Slack #general)
20 Apr 2022	12:06	Service degradation incident email sent out to product owner (Steffie Bosman)
20 Apr 2022	13:01
20 Apr 2022
20 Apr 2022
20 Apr 2022
20 Apr 2022
20 Apr 2022
20 Apr 2022	13:01	Service restored email sent out to product owner (Steffie Bosman)

Proposed Solution

Additional monitoring (Sensu checks) will be added
the issue was solved by disabling Packet Cache on PowerDNS, which is enabled by default: https://docs.powerdns.com/recursor/settings.html#disable-packetcache