Incident description
The Dashboard uses UAT Crowd for authentication and authorisation for historical reasons. When UAT Crowd went down, the Dashboard application was updated to point at Prod-crowd immediately. However, the Prod-crowd was also compromised the same day and couldn't serve authentication and authorisation requests from Dashboard.
Root cause incident report is available at: 2019-July-15 - Crowd Compromise Incident
Incident severity: CRITICAL
Data loss: NO
Timeline
| Time (CET) | |
|---|---|
| 16 Jul, 07:45 | uat-crowd went down, Issue Reported by OC |
| 16 Jul, 08:00 | Fixed by Robert L - By updating Dashboard to point at prod-crowd |
| 16 Jul, 22:30 | prod-crowd compromised. |
| 17 Jul, 12:30 | Changes made by Robert to bypass Crowd by using direct login. |
| 17 Jul, 13:50 | Temoor tested and approved the changes |
| 17 Jul, 14:00 | The local account login method applied to production. |
Proposed Solution
The Dashboard application's authentication method has been updated to bypass Crowd and use local accounts.
Future Mitigation
Internal crowd instance is end of life. The decision was taken to leave Dashboard authentication method to local accounts and not change it back to Crowd. The upcoming Dsahboard V3 will use federated login for authentication and GÉANT CAMS for authorisation.