...

It is crucial to have a trust anchor for all issued client certificates which is stable on the long-term. To that end, an offline hardware-backed CA is provisioned and kept in a physically safe position in GEANT property (TBD: where exactly is it stored, access controls to physical location). The CA itself is created with the CA generation script publicly available on GitHub.

CA operations are performed on the (TBD: project-procured ) Raspberry Pi 3. The Pi needs the following preparatory actions:

• install Raspian Stretch (or higher); required for having openssl 1.1+
• install the package rng-tools (provides access to the built-in hardware random number generator under /dev/hwrng)
• set the date and time (Raspberry Pi does not have a built-in clock)
• after installing all needed packages, remove the Pi from the network and never connect it again.

Q to the SM: is it acceptable to take the preparatory steps before traveling to the signing ceremony? Or do everything live?

...

Afterwards, edit again settings/openssl-rsa.cnf  and settings/openssl-ecdsa.cnf settings with new URLs for the intermediate (Issuing) CA.

...

CA.generateNewIntermediateCA

...

...

...

Immediately after creation, create a new CRL (to assert that there are no revoked certificates at this point in time) and a new OCSP statement for the newly created intermediates:

...

CA.newOCSPStatementForSerial_RSA <serial0x<serial number in decimalhex of the new RSA intermediate certificate>

CA.newOCSPStatementForSerial_ECDSA <serial0x<serial number in decimalhex of the new ECDSA intermediate certificate>

...

The script will be executed by the dev team during initial installation, directly on one of the RADIUS servers (auth-1.hosted.eduroam.org) so that the server certificate private keys are immediately on the right host and need no copying.

...

Copy the server certificates, the private keys and the FreeRADIUS config snippets to the other RADIUS servers in the cluster (auth-2.hosted.eduroam.org).

Store the CA certificate private key set offline in a safe place.

Important: the CA certificates need to contain a valid URL for their CRL Distribution Point. The CRLDP is set by the addnro.py script as: CRLDP.0=http://ocsp.hosted.eduroam.org/rsa/server/<NRO>/crl/root.crl (where <NRO> is the ccTLD of the NRO in question, in capitalised letters - e.g. "PL"(TBD: where, access controls)

Service Operation

Web Service (hosted.eduroam.org)

...

• Apache2
• PHP script for OCSP responses (contained in CAT distribution, utils/ocsp_web/*)

Logs:

• /var/log/apache2/*

Statistics for KPIs

The service statistics are collected with simple SQL queries from several databases.

Number of NROs and IdPs in the system

To be executed on the database "managed_idp" on hosted.eduroam.org. The data is cumulative since start of technical setup of the hosts.

SELECT COUNT(distinct p.inst_id) AS active_inst, i.country AS federation
   FROM silverbullet_user su LEFT JOIN profile p ON su.profile_id = p.profile_id LEFT JOIN institution i ON p.inst_id = i.inst_id 
   WHERE su.deactivation_status = "ACTIVE" 
   GROUP BY federation 
   ORDER BY active_inst DESC;

Example output at service launch day (20 Mar 2019):

+-------------+------------+
| active_inst | federation |
+-------------+------------+
|           6 | LU         |
|           5 | PL         |
|           1 | AM         |
|           1 | CA         |
|           1 | ES         |
|           1 | JP         |
+-------------+------------+

Certificate count (issued/revoked/expired)

To be executed on the database "managed_idp" on hosted.eduroam.org. The data is cumulative since start of technical setup of the hosts.

Total issued certificates

SELECT count(*) AS certcount, ucase(substr(substr(cn,locate('.',cn)+1),1, 2)) AS userfed 
   FROM silverbullet_certificate 
   GROUP BY userfed 
   ORDER BY certcount DESC;

Example output at service launch day:

+-----------+---------+
| certcount | userfed |
+-----------+---------+
|       125 | PL      |
|        60 | LU      |
|        20 | NO      |
|        13 | AM      |
|         4 | JP      |
|         2 | CA      |
|         1 | UA      |
+-----------+---------+

Total revoked certificates

SELECT count(*) AS certcount, ucase(substr(substr(cn,locate('.',cn)+1),1, 2)) AS userfed 
   FROM silverbullet_certificate 
   WHERE revocation_status = "REVOKED" 
   GROUP BY userfed 
   ORDER BY certcount DESC;

Example output at service launch day:

+-----------+---------+
| certcount | userfed |
+-----------+---------+
|        11 | AM      |
|         5 | NO      |
|         5 | PL      |
|         3 | LU      |
|         1 | JP      |
+-----------+---------+

Total expired certificates (certificates which were revoked before they expired are always counted under 'revoked', even after expiry)

SELECT count(*) AS certcount, ucase(substr(substr(cn,locate('.',cn)+1),1, 2)) AS userfed 
   FROM silverbullet_certificate 
   WHERE expiry < NOW() AND revocation_status = "NOT_REVOKED" 
   GROUP BY userfed 
   ORDER BY certcount DESC;

Example output at service launch day:

+-----------+---------+
| certcount | userfed |
+-----------+---------+
|        11 | PL      |
|         7 | LU      |
+-----------+---------+

Authentication count

To be executed on the database "eduroam" on auth-1.hosted.eduroam.org and auth-2.hosted.eduroam.org (each server maintains its own counters). These statistics can't be accumulative because data is deleted after 6 months. So, instead, the queries below return the counters for the last 24h. They should be run on exactly the same time of day every day to maintain full statistics coverage.

Successful authentications

SELECT COUNT(*) AS authcount, SUBSTR(username,LOCATE('@',username)+1) AS
realm FROM eduroamauth WHERE reply = "Access-Accept" AND authdate >=
TIMESTAMPADD(DAY, -1, NOW()) GROUP BY realm ORDER BY authcount DESC;

Example output at day after service launch:

+-----------+-----------------------------+
| authcount | realm                       |
+-----------+-----------------------------+
|       288 | 9-9.lu.hosted.eduroam.org   |
|        54 | 16-15.no.hosted.eduroam.org |
|        40 | 10-10.lu.hosted.eduroam.org |
|         6 | 11-19.lu.hosted.eduroam.org |
|         2 | 23-21.ua.hosted.eduroam.org |
|         1 | 20-20.jp.hosted.eduroam.org |
+-----------+-----------------------------+

Failed authentications

SELECT COUNT(*) AS authcount, SUBSTR(username,LOCATE('@',username)+1) AS
realm FROM eduroamauth WHERE reply = "Access-Reject" AND authdate >=
TIMESTAMPADD(DAY, -1, NOW()) GROUP BY realm ORDER BY authcount DESC;

Example output at day after service launch:

+-----------+-----------------------------+
| authcount | realm                       |
+-----------+-----------------------------+
|        25 | 16-15.no.hosted.eduroam.org |
+-----------+-----------------------------+

Interplay of the eduroam Managed IdP system components

eduroam Managed IdP includes multiple components which need to interwork correctly for the service as a whole to work. The following external dependencies between the components exist

eduroam Managed IdP web frontend → OCSP responder

• issues OCSP statements for each of the certificates known to the system, using a cron job. See documentation on GitHub above. Make sure the cron job is running and verify that updated statements end up in the correct directory on the OCSP responder.

eduroam Managed IdP web frontend → CAT code signing cluster

• web frontend creates installers for Windows, macOS and iOS which are to be digitally signed. The actual signature on the files is offloaded to the existing eduroam CAT code signing cluster (machines located in SURFnet premises). Make sure HTTPS traffic from the web frontend to the signing cluster is allowed.

eduroam Managed IdP RADIUS Server → OCSP responder

• makes request at OCSP responder during every user authentication. Make sure the HTTP communication between RADIUS server and OCSP Responder is possible.

Interplay of the eduroam Managed IdP system components

eduroam Managed IdP includes multiple components which need to interwork correctly for the service as a whole to work. The following external dependencies between the components exist

eduroam Managed IdP web frontend → OCSP responder

• issues OCSP statements for each of the certificates known to the system, using a cron job. See documentation on GitHub above. Make sure the cron job is running and verify that updated statements end up in the correct directory on the OCSP responder.

eduroam Managed IdP web frontend → CAT code signing cluster

• web frontend creates installers for Windows, macOS and iOS which are to be digitally signed. The actual signature on the files is offloaded to the existing eduroam CAT code signing cluster (machines located in SURFnet premises). Make sure HTTPS traffic from the web frontend to the signing cluster is allowed.

eduroam Managed IdP RADIUS Server → OCSP responder

• makes request at OCSP responder during every user authentication. Make sure the HTTP communication between RADIUS server and OCSP Responder is possible.SELECT COUNT(*) AS authcount, SUBSTR(username,LOCATE('@',username)+1) AS realm FROM eduroamauth WHERE reply = "Access-Accept" AND authdate >= TIMESTAMPADD(DAY, -1, NOW()) GROUP BY realm ORDER BY authcount DESC;+-----------+-----------------------------+ | authcount | realm | +-----------+-----------------------------+ | 288 | 9-9.lu.hosted.eduroam.org | | 54 | 16-15.no.hosted.eduroam.org | | 40 | 10-10.lu.hosted.eduroam.org | | 6 | 11-19.lu.hosted.eduroam.org | | 2 | 23-21.ua.hosted.eduroam.org | | 1 | 20-20.jp.hosted.eduroam.org | +-----------+-----------------------------+