Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Incident description

The Dashboard uses UAT Crowd for authentication and authorisation for historical reasons. When UAT Crowd went down, the Dashboard application was updated to point at Prod-crowd immediately. However, the Prod-crowd was also compromised the same day and couldn't serve authentication and authorisation requests from Dashboard.

Root cause incident report is available at: 2019-July-15 - Crowd Compromise Incident

Incident severity: CRITICAL

Data loss: NO

Timeline


Time (CET)
16 Jul, 07:45uat-crowd went down, Issue Reported by OC
16 Jul, 08:00

Fixed by Robert L - By updating Dashboard to point at prod-crowd

16 Jul, 22:30

prod-crowd compromised.

17 Jul, 12:30

Changes made by Robert to bypass Crowd by using direct login.

17 Jul, 13:50Temoor tested and approved the changes
17 Jul, 14:00The local account login method applied to production.

Proposed Solution

The Dashboard application's authentication method has been updated to bypass Crowd and use local accounts.

Future Mitigation

Internal crowd instance is end of life. The decision was taken to leave Dashboard authentication method to local accounts and not change it back to Crowd. The upcoming Dsahboard V3 will use federated login for authentication and GÉANT CAMS for authorisation.