...
Additions to metadata best current practices need to be decided by the eduGAIN SG. Each such good practice needs to needs to be implemented as an eduGAIN validator warning by the eduGAIN OT. Each good practice rule needs to be implemented in [eduGAIN-BCP].
System maintenance
System updates
- All virtual machines running eduGAIN services are regularly updated.
- Before an update is planned, the local personel at PSNC are notified in the case of an update failure and immediate restore. An update forward notice is sent to the eduGAIN SG.
- In the case of large configuration changes, like moving services to new hosts, applying large infrastructure changes etc., a notice at least 7 days in advance is sent to the eduGAIN SG
- All changes are documented in the log available for inspection at: https://technical.edugain.org/system_updates
Backups
Metadata aggregation procedures
The technical details of the aggregation process are described in [eduGAIN-meta]. Here we only present the operational implementation of this process.
The aggregation, signing and publishing of the eduGAIN metadata aggregate is done on the hourly basis
All information about the system status, federation metadata channel information, federation public keys etc. is kept in the eduGAIN database and taken from there as required within the aggregation process.
Half past every hour metadata acquisition is started on mds-feed and is pefromed in the following steps
mds-feed downloads federation metadata feeds using conditional GET.
if the conditional GET resulted in a download of a new metadata file, such file is passed through the local validator instance, if validation succeeds the downloaded file is used as an input for aggregator if it fails, the previous correct feed copy us used instead
- the newest available validated copy of the federation metadata feed is kept for future use
the validated metadata files are passed to a pyFF flow, see also [eduGAIn-meta] Metadata combination and collision handling
pyFF aggregates and then signs the resulting feed
the resulting file is analysed, broken into entities and used to update the edugain-db
the final output is uploaded with sftp to the technical host using a dedicated user account on the the technical host
at 45 minutes past every hour the new copy of eduGAIN metadata aggregate is copied to the final destination directory and when the copy is completed the mv action is performed in order to substitute the production file in an atomic mode
finally the new eduGAIN metadata aggregate file is copied to the history repository and compressed
Metadata aggregation related procedures
The technical details of the aggregation process are described in [eduGAIN-meta]. Here we only present the operational implementation of this process.
Aggregation, signing and publishing
The aggregation, signing and publishing of the eduGAIN metadata aggregate is done on the hourly basis
All information about the system status, federation metadata channel information, federation public keys etc. is kept in the eduGAIN database and taken from there as required within the aggregation process.
Half past every hour metadata acquisition is started on mds-feed and is pefromed in the following steps
mds-feed downloads federation metadata feeds using conditional GET.
if the conditional GET resulted in a download of a new metadata file, such file is passed through the local validator instance, if validation succeeds the downloaded file is used as an input for aggregator if it fails, the previous correct feed copy us used instead
- the newest available validated copy of the federation metadata feed is kept for future use
the validated metadata files are passed to a pyFF flow, see also [eduGAIn-meta] Metadata combination and collision handling
pyFF aggregates and then signs the resulting feed; currently the signing is done with key files stored at the mds-feed host
the resulting file is analysed, broken into entities and used to update the edugain-db
the final output is uploaded with sftp to the technical host using a dedicated user account on the the technical host
at 45 minutes past every hour the new copy of eduGAIN metadata aggregate is copied to the final destination directory and when the copy is completed the mv action is performed in order to substitute the production file in an atomic mode
finally the new eduGAIN metadata aggregate file is copied to the history repository and compressed
At midnight (CET) hourly copies of metadata are deleted from the repository, At midnight (CET) hourly copies of metadata are deleted from the repository, leaving only a single daily file. These daily files can then be used as a source of various data analysis.
...
One example of such a special situation may be a real case of an introduction of a valid but a very short-lived metadata file followed by a metadata error causing a aggregation reject. That situation, leaving no time for normal procedures to take place, was caused by a configuration error on the federation side and was rectified in a short time while the eduGAIN aggregation was suspended.
System maintenance
System updates
- All virtual machines running eduGAIN services are regularly updated.
- Before an update is planned, the local personel at PSNC are notified in the case of an update failure and immediate restore. An update forward notice is sent to the eduGAIN SG.
- In the case of large configuration changes, like moving services to new hosts, applying large infrastructure changes etc., a notice at least 7 days in advance is sent to the eduGAIN SG
- All changes are documented in the log available for inspection at: https://technical.edugain.org/system_updates
Backups
- system backups are performed daily as a part of the standard PSNC backup routine
- virtual machine snapshots are performed prior to system updates
- four times a year a full virtual machine dump is performed
Disaster recovery
In the case of an unexpected problem resulting from metadtata aggregation (which may result from an unusual error on a federation side or some software bug in one of the aggregation process steps) the eduGAIN OT has access to hourly copies reaching 24 hours back and to several years of daily copies .
Restoration of snapshots or full virtual machines is possible (and has been performed several times not as disaster recovery but in order to get access to some old files for statistics reasons).
Technical details
eduGAIN database description
...