Page tree
Skip to end of metadata
Go to start of metadata

This is a working document, major changes can be expected at any time.

Introduction

This document describes operational procedures implemented to support eduGAIN SAML services. It is required by the eduGAIN SAML Profile document [eduGAIN-Profile] and in addition to the Metadata Aggregation Practice Statement must be seen as complementary to eduGAIN SAML Profile.

eduGAIN services

Under the term services listed are utilities as perceived by external users. The internal organisation of services, flow of information and dependencies are not relevant in this view, but are described in sections further down.

Core Services

NameAccess locationDescriptionManaged by
MDShttps://mds.edugain.orgeduGAIN Metadata Distribution Service (MDS) is the central component of the eduGAIN service as a whole. For the  detailed description and procedures used in the eduGAIN metadata aggregate distributed by MDS see [eduGAIN-meta]. The eduGAIN metadata aggregate is produced on a separate, secured host (mds-feed) and it is copied to the distribution hosts and served form there by the http server. The file is updated hourly.OT
The technical sitehttps://technical.edugain.orgThe technical site is directed primarily at the federation level thechnical personel. It provides information about eduGAIN members, details about their participation. The technical site is also the distribution point of documentation and the home for several core and supplementary services.OT
Validatorhttps://validator.edugain.orgThe eduGAIN validator is a service designed for validating metadata adherence to standards and eduGAIN requirements. The software has been created primarily as a component for the eduGAIN metadata aggregation and the details of validation rules are given im [eduGAIN-meta]. The same software enriched by a GUI is used as a tool for manual validation of metadata and serves as a support tool for federation operators.OT
eduGAIN status informationhttps://technical.edugain.org/statusThis status page provides a view of the eduGAIN database in the part relevant to membership information and the current status of metadata aggregation. The page also displays short summary information about numbers of entities in eduGAIN. The interface provides links to scans of the eduGAIN declaration documents signed by federations, direct links to metadata validation, links to contacts, metadata sources etc.OT
Entities database GUIhttp://technical.edugain.org/entiesThis service is an interface to the part of the eduGAIN database which stores information about entities themselves. The interface has many filtering mechanisms and also allows for CSV download for further processing in a spreadsheet.OT
eduGAIN database APIhttps://technical.edugain.org/apiThe API provides access to most of information stored in the database. In particular, the API may be used by the federations to monitor the eduGAIN aggregation process. Other uses are statistics of various sorts or even download of membership maps.OT

Supplementary services

NameAccess locationDescriptionManaged by
ECCShttps://technical.edugain.org/eccs/eduGAIN Connectivity Check Service is a monitoring service for IdPs listed in eduGAIN, testing if they are actually ready for eduGAIN, i.e. if they consume eduGAIN metadataOT
isFederated Checkhttps://technical.edugain.org/isFederatedCheck/This tool searches all known academic identity federations for matching organisations and then displays the results.OT
CoCo monitorhttp://monitor.edugain.org/coco/Monitoring service testing for REFEDS Code of Conduct complianceSRCE
Technical testing platformhttp://technical-test.edugain.orgThis host serves as a playground for software development done by the operational team. All extensions are applied, tested and presented at this platform and then transferred to production using the git mechanismOT
WIKI
The WIKI is maintained as a part of the GEANT WIKI space. The content is provided by many members of the community. WIKI serves as technical documentation, formal documentation (meeting minutes, documentation of operational procedures) and various guides on joining and making most of eduGAINGEANT core
Support


Operational Team tasks

The team

As defined in [eduGAIN-CONST] the Operational Team (OT) is responsible for:

  • Daily technical issues in central eduGAIN operations (e.g. website, central member database).
  • Collaboration with the operators of each Technology Profile.
  • Receiving enquiries about eduGAIN and forwarding them to the appropriate body.
  • Receiving, reviewing and processing applications to join eduGAIN against basic eligibility criteria as set out in the eduGAIN Policy Declaration.
  • Preparing and publishing an eduGAIN Operational Practice Statement (this document) for the eduGAIN interfederation service, covering central operations and relationship with each technology profile.
  • Preparing an audit plan for the eduGAIN operational practices on the request of the eSG.

At the moment the OT also acts as the operator of the SAML technology profile.

Management of core eduGAIN services

  • management of virtual machines (access management, system maintenance - installation and updates, global backups, status monitoring)
  • management of eduGAIN core services (maintenance of any software tools required by the services, monitoring of services, specialised backups)
  • supervision of the aggregation function - reacting to aggregation errors, supporting federations in location of problems
  • technical documentation - maintenance of user documentation of eduGAIN services
  • user support - done in cooperation with the eduGAIN support team
  • management of the development platform (based on the GEANT git)
  • service development - configuration changes and extensions of existing services, in particular any development work within the eduGAIN MDS, validator, database

Management or supervision of supplementary eduGAIN services

  • eduGAIN OT directly manages:
    • ECCS
    • isFederated check
  • eduGAIN OT supervises
    • CoCo monitor
    • WIKI

Supervision of eduGAIN joining process

The task of chairing the eduGAIN SG lies within the Operational Team. The SG chair supervises the joining process of new members, sets up consultations, handles voting, keeps the documentation of the process. As a part of the joining process federations are required to provide contact and technical information including sensitive factors such as public keys used for signature verification.

eduGAIN operational model and availability of services

eduGAIN core function is the metadata exchange point. Federations supply their own metadata and download aggregated metadata to supplement their own and redistribute them within their federation members. Federations are strongly discouraged from pointing any of their members directly to the eduGAIN MDS. Within this operational model even a relatively long (several hours) downtime of the MDS does not cause any disruption that could be noticed by individual identity or service providers.

While every care is taken that all eduGAIN services function reliably, the selected operational model allows that services updates and modifications can be done at a short-term notice allowing for a small risk of a downtime required to restore the system snapshot.

Operational Team procedures

Registration and modification of SAML profile related federation information

information typesecurity level
federation SAML policy URL1

registration practice statement URL

1

federation SAML metadata aggregate access URL

3

federation metadata signing key

4
registrationAuthority attribute value3

Security levels
security leveldescription
Sspecial - delegating representatives requires contact with the federation management
1informational, not requiring special vetting
2important contact information (while not currently used it may be introduced in the future)
3information of eduGAIN operational relevance, requires special care
4crucial for eduGAIN trust, requires utmost care

Introduction of new requirements for federation metadata feeds

As a principle requirements for federation feeds must be based on either general standards to which eduGAIN SAML profile adheres or on the eduGAIN SAML profile. In the case of standards, the experience shows that certain violations are only discovered when reported by participating federations - not all such violations are reported by standard schema validation tools, ot in fact not all are just schema errors. Whenever a new problem is reported, the OT makes an assessement whether it inf fact violates a required standard and if so then:

  • the OT implements a new validator rule initially as a warning;
  • The OT informs the SG about adding a nev validation rule together with an assessement of which federations may be affected by it and suggests a grace period, after which the new rule will start generationg an aggregation error;
  • SG members will be given the opportunity to request a longer time-frame, and eduGAIN Support will work with any participants that are currently breaching this requirement to fix the issues before the grace period ends.

Every rule is documented in the [eduGAIN-meta] .

When raising an error, the validator points to the specific rule in  [eduGAIN-meta] .

Introduction of new best current practices for federation metadata feeds

Additions to metadata best current practices need to be decided by the eduGAIN SG. Each such good practice needs to needs to be implemented as an eduGAIN validator warning by the eduGAIN OT. Each good practice rule needs to be implemented in [eduGAIN-BCP].

Metadata aggregation related procedures

The technical details of the aggregation process are described in  [eduGAIN-meta]. Here we only present the operational implementation of this process.

Aggregation, signing and publishing

The aggregation, signing and publishing of the eduGAIN metadata aggregate is done on an hourly basis.

All information about the system status, federation metadata channel information, federation public keys etc. is kept in the eduGAIN database and taken from there as required within the aggregation process.

  • Half past every hour metadata acquisition is started on mds-feed and is pefromed in the following steps:
    • mds-feed downloads federation metadata feeds using conditional GET.

    • if the conditional GET resulted in a download of a new metadata file, such file is passed through the local validator instance, if validation succeeds the downloaded file is used as an input for aggregator if it fails, the previous correct feed copy us used instead

    • the newest available validated copy of the federation metadata feed is kept for future use
    • the validated metadata files are passed to a pyFF flow, see also [eduGAIn-meta] Metadata combination and collision handling

    • pyFF aggregates and then signs the resulting feed; currently the signing is done with key files stored at the mds-feed host

    • the resulting file is analysed, broken into entities and used to update the edugain-db

    • the final output is uploaded with sftp to the technical host using a dedicated user account on the the technical host

  • At 45 minutes past every hour the new copy of eduGAIN metadata aggregate is copied to the final destination directory and when the copy is completed the mv action is performed in order to substitute the production file in an atomic mode

  • Finally the new eduGAIN metadata aggregate file is copied to the history repository and compressed

  • At midnight (CET) hourly copies of metadata are deleted from the repository, leaving only a single daily file. These daily files can then be used as a source of various data analysis.

Handling of aggregation alerts

As described in [eduGAIN-meta], under certain conditions aggregation alerts are raised. The current practice is that these alerts are sent as e-mails to the eduGAIN OT. Since alerts are relatively rare and federation metadata feeds can be cached for at least 4 days, the eduGAIN OT makes its own decision on how to react to a particular alert. Usually before sending a notification to a federation, the OT waits until the next aggregation run to make sure that the situation has not been rectified by the federation.

If the OT decides that a notification needs to be sent to the federation, the official contact address registered be the given federation is used. If one of the following conditions is met:

  • there is a immediate reject form the federation mail server,
  • there is no reply within 24 hours,
  • the metadata validUntil period is under 14 hours

then the OT sends additional notifications to the federation delegate and the deputy using their e-mail addresses registered by the given federation.

If it becomes likely that a given federation may not be able to react in time and that its cached metadata feed may expire, the eduGAIN OT sends a warning to the eduGAIN SG mailing list.

It must be realised that that the case of all entities supplied by a large federation being deleted form eduGAIN has heavy consequences - other participating federations will naturally have to drop these entities. When the federation metadata feed becomes available again, other federations may be forced into running emergency regeneration of their metadata, service providers may observe limited breaks in their service. Therefore the eduGAIN OT is making all possible effort to avoid such situations. If the eduGAIN OT realises a very special situation it is allowed to temporarily stop aggregation in order to avoid the deletion of of the federation but ir MUST immediately notify the eduGAIN SG that such measures have been taken.

One example of such a special situation may be a real case of an introduction of a valid but a very short-lived metadata file followed by a metadata error causing a aggregation reject. That situation, leaving no time for normal procedures to take place, was caused by a configuration error on the federation side and was rectified in a short time while the eduGAIN aggregation was suspended.

System maintenance

System updates

  • All virtual machines running eduGAIN services are regularly updated.
  • Before an update is planned, the local personel at PSNC are notified in the case of an update failure and immediate restore. An update forward notice is sent to the eduGAIN SG.
  • In the case of large configuration changes, like moving services to new hosts, applying large infrastructure changes etc., a notice at least 7 days in advance is sent to the eduGAIN SG
  • All changes are documented in the log available for inspection at: https://technical.edugain.org/system_updates

Backups

  • system backups are performed daily as a part of the standard PSNC backup routine
  • virtual machine snapshots are performed prior to system updates
  • four times a year a full virtual machine dump is performed

Disaster recovery

In the case of an unexpected problem resulting from metadtata aggregation (which may result from an unusual error on a federation side or some software bug in one of the aggregation process steps) the eduGAIN OT has access to hourly copies reaching 24 hours back and to several years of daily copies .

Restoration of snapshots or full virtual machines is possible (and has been performed several times not as disaster recovery but in order to get access to some old files for statistics reasons).

Technical details

eduGAIN database description

The eduGAIN database is central to all eduGAIN core services. The database stores:

  • general and contact information about participating and candidate federations
  • operational information about participating and candidate federations like metadata URLs, signing keys, registrationAuthority values
  • operational information about the metadata aggregation process including details about metadata acquiring form participating federations, results of metadata validation, cache timers for individual participant federations
  • operational information about entities published through eduGAIN derived from the metadata
  • statistics derived from metadata aggregation, like numbers of entities published by individual federations and much more
  • information collected from supporting monitoring services like ECCS, CoCo

The database is placed on a host separated from the external network, accessible only trough a limited numbers of secure hosts. Database access is realised via dedicated user accounts with access right crafted to minimize the possibility of unauthorized changes.

The database is managed mostly via a web interface secured with access passwords. Modification of data on security levels S, 1, 2 can be done without any additional protection. Management of data with security level 3 is protected with on-time passwords mailed to an external mail account of the managing administrator. Management of data with security level 4 requires direct access to the database host. 

For security reasons singing keys can be present only for federations which have been approved to be a member of the eduGAIN SAML Profile.

eduGAIN Metadata Distribution Service (MDS) 

eduGAIN Metadata Distribution Service (MDS) is the central component of the eduGAIN service as a whole. For the  detailed description and procedures used in the eduGAIN metadata aggregate distributed by MDS see [eduGAIN-meta] and theAggregation, signing and publishing subsection within this document. The eduGAIN metadata aggregate is produced on a separate, secured host (mds-feed). Metadata signing is also peformed on mfs-feed currently with a key file located on the host istself.

In order to minimise risks of exposing a high permissions account on the mds host the resulting aggreagate file is transferred from mds-feed to the mds host using a dedicated low premissions account. The aggregate is then moved to the final place on the mds host in a process innitiated within the mds host.

Organisation and management of services

Main access host - technical, validator, mds
DNS names

www.edugian.org, technical.edugain.org; validator.edugain.org; mds.edugain.org

All these are CNAMEs for massonia.man.poznan.pl

Function

eduGAIN database - edugain-db

Functionstore all data for services directly managed by the eduGAIN OT
The aggregation host - mds-feed
Functionacquire and validate federation metadata feeds, create, sign and publish the  eduGAIN metadata aggregate.

Security considerations

The security of the eduGAIN SAML services is essentially the security of the eduGAIN aggregate. This in turn depends on:

  1. validation of federation metadata input data - their originality and integrity - this depends on the safety of federation certificates (stored in the database) and the safety of the signature verification process itself
  2. aggregation process - it is crucial that the resulting aggregate contains exactly the data provided by participating federations (after modifications described in the [aggregation])
  3. aggregation signature - the eduGAIN signing key and the signing process are the key factors here














  • No labels

5 Comments

  1. Supervision of eduGAIN joining process

    Edit suggestions for this section.  The OT currently happens to chair the eduGAIN SG but there is nothing in the constitution that says they have to and this has just been decided on an arbitrary rolling platform.  I suggest we need an SG ToR or to properly complete the joining process section of this wiki and reference that here.

  2. eduGAIN operational model and availability of services

    Edit suggestions for this section.  Typo discoursed / discouraged - although this should probably say MUST NOT under new guidelines.  I would also probably expand on why we discourage directly consuming from MDS in that trust is established via the resigned version from the local federation operation and consuming directly breaks that trust model.

  3. Registration and modification of federation information

    typo last word of section "profile".

  4. Introduction of new requirements for federation metadata feeds

    As discussed this week - I don't think that the SG should "decide" or "vote" on an implementation when a violation is discovered as this gives the impression that they have some sort of choice to manage violations differently and allow certain violations.  I would change this to say then when a process for managing a violation has been identified and an appropriate mechanism added to the validator, the Operational Team will inform the SG and set out an appropriate grace period before this is implemented as a violation.  SG members will be given the opportunity to request a longer time-frame, and eduGAIN Support will work with any participants that are currently breaching this requirement to fix the issues before the grace period ends.

  5. Introduction of new best current practices for federation metadata feeds

    Just to note we can hopefully add some more information here as we flesh the BCP process out.