Page tree

78

Cost Benefit Analysis

(CBA)

 

Purpose:

 

A cost-benefit analysis (CBA) is an analytical tool for assessing the advantages and disadvantages of moving forward with a business proposal or project. By using a cost-benefit analysis template, Activities can identify quantitative and monetary estimates to determine whether to pursue an initiative, tweak it or abandon it completely.

This document is your template for producing a GEANT cost benefit analysis. The CBA is created for the GEANT PMO by a project sponsor, department or unit seeking funding, approval, or both for an activity, initiative, or project.

 

Created by:

The CBA is created and maintained by the GEANT Activity Leader.

 

Submit to:

projectdocumentation@geant.org (the GÉANT Project Office).

 

CBA Information

Project Name:

eduGAIN Support

CBA Author:

Ann Harding, Thomas Baerecke, Lukas Haemmerle, Justin Knight, Marina Adomeit

CBA Code:

 

Project Type/Size

Medium

Date submitted

 

Activity (if applicable)

GN4-2 SA2 and JRA3

Task(if applicable)

T2  

Production Gate Approval Meeting

Decision:

 

Date of Meeting:

 

Comments:

 

 

Table of Contents

 


1                      Executive Summary

The eduGAIN service interconnects identity federations around the world, simplifying access to content, services and resources for the global research and education community. Through eduGAIN, identity providers offer a greater range of services to their users as delivered by multiple federations in a truly collaborative environment; service providers offer their services to users in different federations thereby increasing their target market; and users seamlessly benefit from a wider range of services.

The eduGAIN interfederation service delivers a platform for the trustworthy exchange of metadata through the coordination of technical infrastructure and policy. This supports the needs of federations in establishing a common baseline for metadata interoperability and furthers the goals of federations to operate in a global identity access and service exchange.

Prior to the pilot of the eduGAIN Support Service, operational activities and support have been strictly limited to those related to the MDS - the central metadata service of eduGAIN, and supporting federations in correctly managing their metadata feeds to and from this system. The growing demand from research communities and from a set of federations to extend eduGAIN support beyond this scope lead to the development of the eduGAIN Support Service pilot, to provide support and coordination for multi-party troubleshooting of performance affecting issues.

The pilot, that on approval from GEANT PLM started on 1st March 2017, was reviewed as successful and the transition to production was approved under the GEANT PLM process on 5th January 2018. This is updated CBA to be presented for the enter production approval gate, and it will be valid during the eduGAIN support production operations in SA2 and inherited in GN4-3. Alongside this CBA, roadmap and relevant operational documentation have also been updated in support of the recommended action.

1.1                Summary of work to date

Work began on eduGAIN Support at the beginning of GN4-2, with the analysis of the then-current support channels for eduGAIN and gathering of intelligence around the requirement to enhance this. Under Task 2 of JRA3, a team of federation experts and supporting professionals designed a recommended service to take into pilot, which was approved by the GEANT PLM team on 1 st March 2017.

A part of the pilot, a dedicated ticket queue on the GEANT ORTS ticketing system was requested and configured, at zero cost (except some time) to the Task owing to GÉANT already having this availability. This ticketing queue was used to manage the support queries raised during the pilot and found fit-for-purpose. A rota of 2 people per week was established to manage queries, which was also found to be a suitable level of support availablility, with each person spending in the order of 2 hours per week addressing tickets raised. The team not only responded to support requests and questions but also proactively helped reduce potential issues that could arise due to incorrect eduGAIN metadata. This was done by relying on external checking tools (mostly operated by UK federation) and by actively informing federation operators of the issues.

On approval of the GEANT PLM team on 5 th January 2018,  the transition of the service to production to the SA2 eduGAIN service operations in Task 2 started. The transition timeline was defined and the releavant individuals from JRA3 and SA2 were assigned to carry out the roadmap tasks. During the transition, the manuals, workflow and documentation for the support team were finalised. The scope of the eduGAIN support was additionally agreed and synchorinzed with the eduGAIN service manager. Two main areas of the support team were reinforced: the reactive work - when the support team responds to the received queries; and proactive work - when the support team reacts on potential errors within the identity federations domain and take proactive actions to resolve them. The scope of the proactive work will be adapted in collaboration with the service manager as normal ongoing activities in order to reflect the correct eduGAIN functions and needs of maturing the eduGAIN service adoption. The structure of the team was defined, assigning the lead of the support team with defined responsibilities and the rota of three support team members.  Finally, the eduGAIN support channel was published on eduGAIN websites, an d legacy channels were removed form the public sites: https://edugain.org/contact/   and   https://technical.edugain.org/contact .

1.2                Recommendation

This Cost Benefit Analysis therefore recommends entering into a production service of eduGAIN Support,   enhancing the staffing and scope of work of the central eduGAIN support team to support the coordination, troubleshooting, resolution and follow-up of complex multi-party performance or operational failures which involve multiple parties in eduGAIN.

The operational documentation created during the transition process and based on the Service Design accompanies this CBA and is awailable in the internal service operations wiki space.

1.3                Supporting Reasons

The reasons for this recommendation are that the results of the pilot in increasing in the size of the operational team of eduGAIN to support more complex troubleshooting delivers the benefits summarised in Section 4 : Benefits / Impacts .

1.4 Summary Information

The cost of production will be fully funded by the GEANT project to cover the 0,4 FTE for the support team lead and support rota members and is estimated to €36,600 annualy based on the average FTE cost in the GEANT project. The benefit is measured in terms of time and effort saved to the community, detailed below.

Time saving on coordination work, specialist knowledge and communication overhead for research SPs and other similar SPs with a wide user footprint i.e. with users at IdPs in 5 or more countries.

  • In simplest case (est. 10% of cases), 20 minutes in contacting a range of IdPs, assuming all communication channelled by federation OR only one site per country impacted - c30 hours.
  • In majority case (est. 85%), 20 minutes in contacting a range of IdPs, assuming all communication channelled by federation OR only one site per country impacted – c50 hours.
  • In extreme cases (estimated 5%) save 4 days [1] - c.160 hours

Time saving on coordination work and communication overhead for federations where there is an incident impacting more than 2 entities:

  • In simplest case (est. 10% of cases), 5 mins per federation in contacting more than one other impacted party: c8.3 hours
  • In majority case (est. 85%), save 30 mins: c42 hours
  • In extreme cases (estimated 5%) save 1 day: c 40 hours
  • Total estimated saving per 100 cases that include > 2 impacted sites: just over 2 weeks, or EUR5000

Recommended Approach – Summary information

Investment Information

Investment Costs

Production cost over 1 year

36,600

GN4-2 SA2 Funding

9,150

GN4-3 Funding

146,400

Total Funding for GN4-2 and GN4-3

155,550

Technical Annex Reference (if applicable)

GN4-2-SA2 T2 for operations

GN4-3-WP5 T1 for operations

 

2                      Phase 1. Background Information

2.1                Organisational Overview

On a day-to day level, the eduGAIN interfederation service consists of two main elements:

        eduGAIN Policy Framework.

        Metadata Distribution Service (MDS).

The eduGAIN Policy Framework details the administrative and technical standards that all participant federations must adhere to in order to enable the trustworthy exchange of service information to support identity, authentication and authorisation between partner federations. It is administered by the eduGAIN Product Manager, who holds the role of moderator for the eduGAIN Steering Group (SG). The eduGAIN SG are representatives of the member federations of eduGAIN and are responsible for the technical governance of the service.

The key stakeholders for eduGAIN are they direct members who have specific roles and responsibilities in governing the technical service and carry out specific tasks in order to participate

  • National identity federations in GÉANT who are members of eduGAIN
  • Global identity federations

Prior to the eduGAIN Support pilot, formal support activities of eduGAIN were targeted only at these groups.

The following additional groups benefit significantly from eduGAIN or provide benefit to eduGAIN but are a step removed, via identity federations.

  • Research communities
  • Campuses
  • Commercial services

For the majority of these groups, eduGAIN is, by design, an invisible service, an additional functionality offered by their national federation. The eduGAIN Support pilot addressed how eduGAIN engages with them and in particular, a subset which are active primarily at an interfederation-only level and are aware of eduGAIN and contact eduGAIN addresses directly, despite pre-existing contracts with different federations and support channels available to them. Prior to the pilot, the only method of dealing with these mails was a polite note that we cannot help, and directing them to a database to look up IdP contacts individually, or to their respective federation who may do the same, or in some cases, more. This created a perceived lack of consistency and value in eduGAIN for these communities, despite the system overall working well when queries land in the most appropriate place. Through the eduGAIN Support Service, these groups, as well as national and global identity federations have a single point of contact for eduGAIN support, with a team of experts working on rota providing both local and international expertise and co-ordination between affected parties.

2.1.1       Community Need

Prior to the eduGAIN Support Service pilot, support queries for eduGAIN came in via varying channels. People contacted the eduGAIN operations team (OT), which normally handles technical issues concerning day-to-day operations. All other queries were forwarded somewhere else, such as the REFEDS (the Research and Education FEDerations group) mailing list, which might channel them back to someone else, including the eduGAIN OT. Via personal relationships within the larger eduGAIN community some people’s questions did arrive at their established contact, however this was at best informal and provided no means of measuring the level of support required in a centralised manner. Observing these inefficient and diffuse methods, the requirement for a uniform helpdesk as single point of contact for all eduGAIN-directed queries was identified. The eduGAIN Support Service meets this requirement and serves both federations and SPs who have knowledge of eduGAIN.

Aggregating user questions in one single place also enables the federation operators to have a global, aggregated view of the number and scope of eduGAIN-specific questions coming in, making it easier to create a useful knowledge base and hence limiting the effort required to support eduGAIN users both centrally and directly at federations.

2.1.2       Drivers for Change

The research community, including CERN and LIGO have expressed significant doubts about the fitness of eduGAIN for their needs without a more active central operations team to support the more complex uses of federated identity that they require.

Federations are not all equally capable of bearing the burden of long-lasting and complex multi-party troubleshooting.

Federations that are able to carry this workload would still benefit by being freed up from coordination work to focus more closely on their directly involved users and customers.

Future funding for eduGAIN operations from the European Commission is likely to be strongly driven by the impact eduGAIN has on research and its use within research communities. In order to secure the funding that exists for the bare-bones version of eduGAIN Operations, it is therefore strategically important to invest in this area.

2.1.3       Outcomes

The outcomes of eduGAIN support will be:

  • eduGAIN support enhanced to support performance-related incidents in interfederation
  • An aggregation of the many forms of contacting eduGAIN into a single ticketing system
  • Processes and workflows for coordinating performance incidents that have an interdomain scope
    • Processes will ensure the role of the federation operator is engaged and will not attempt to replace it centrally
  • A staffed rota of responders, delivered by federation experts
  • A KB/FAQ developed on an on-going basis and made available to the community
  • A full SD service during production:
    • Delivered by a collaboration of GÉANT SD, GÉANT T&I experts and federation experts.

The eduGAIN support team will be available to:

  • Federations who are troubleshooting on behalf of their IdPs or SPs
  • IdPs or SPs who have somehow found their way to eduGAIN independently with interfederation problems or queries

The main working tool for the eduGAIN support team is the ticketing system. The requirements were identified for the system and an instance of GEANT’s OTRS queue was deployed, tested and found fit-for-purpose during the pilot and will continue to be used in the production service.

2.1.4       KPIS

For the production service, this CBA recommends two KPIs for the number of cases handled; reactive tickets and pro-active tickets:

Reactive tickets :

  • Production 2018: 20 (production starts by end of October 2018)
  • Production 2019: 150
  • Production 2020: 125

Proactive tickets (e.g. metadata clean-up, checking of error logs, identifiying zombie IDs):

  • Production 2018: 10 (production starts by end of October 2018)
  • Production 2019: 40
  • Production 2020: 30

3                      Description Of Alternatives Considered

3.1                Option 1 – Do Nothing

eduGAIN could continue with the current model and address these e-mails on an ad hoc basis, simply directing them to a different contact as appropriate.

While this would continue to keep the operational costs low, and addresses c2/3 of the use cases in eduGAIN (evaluated based on an analysis of the types of SP in eduGAIN), the remaining 1/3 consists of the complex research communities that the EC wishes e-Infrastructures to support. If eduGAIN is to make a case for continued funding of operations by the EC into GN4-3, these cases have to be visibly and enthusiastically supported, even if it were not already fundamentally compatible with the GÉANT strategy to do so.

In the event that the 'do nothing' option is chosen, GÉANT will need to look at alternative models of funding eduGAIN (e.g. by subscription) in the event that EC funding is tied to delivery of service to researchers. Even with this option, it is then likely that key research groups would collaborate to build an e-research specific alternative to eduGAIN as the technology aspects are simple to recreate, and users would then ultimately suffer by having to maintain multiple accounts.

This option was already discarded at the enter production gate review by GEANT PLM on 5 TH January 2018.

3.2                Option 2 – Full First and Second Line support, with GÉANT SD

This is the option that has been piloted and was recommended and approved by GEANT PLM to transition to production. It combines the infrastructure and availability of service teams in GÉANT with the know-how of federation operators.

Details are in the section 2.1.3 on 'Outcomes'.

3.3                Option 3 – Manage response with distributed SLAs

Harmonise the researcher experience of eduGAIN by adopting harmonised support with SLAs.

The experience with attempting to deliver multi domain services involving SLAs by NREN partners over the last 10 years has predominantly been unsuccessful. The ability of organisations to commit is not unified, and the change from social to contractual trust can result in loss of service, rather than an increase. [2]

At a federation level, not every federation has the depth of staff or scope of service to address complex e-Research needs. The 2016 REFEDS survey [3] (slide 5) indicates a wide gap between federations with significant investment and those with minimal manpower and budget. In addition, the primary business driver for almost every national identity federation is to serve the needs of the campus, rather than the service providers who are the most frequent contactors of eduGAIN centrally. This means that an approach where eduGAIN would recommend an SLA-type of arrangement via federations to serve these use cases would not be viable, even if it were acceptable to the community.

This option was already discarded at the enter production gate review by GEANT PLM on 5 TH January 2018.

4                      Benefits / Impacts

Benfits of the eduGAIN Support Service include:

  • Time saving for federations in coordination overhead
  • Better user experience as issues are resolved more quickly and information shared more frequently and efficiently between affected parties.
  • More professional image for eduGAIN as a whole, especially among the e-Science users who do have direct knowledge of the eduGAIN brand, unlike more traditional users.
  • Better knowledge and experience sharing with the wider community as a KB of experience will be created and made available.
  • Improved ability to engage collectively with vendors on bugs and feature issues as a greater picture of impact will be available.
  • Increased value perception of eduGAIN to national and global identity federations, as well as research groups, driving adoption of eduGAIN and thereby greater service access globally
  • Improved metadata within eduGAIN, making issues less frequent and the overall eduGAIN service cleaner and more efficient to users
  • Assistance for SPs when unclear with which eduGAIN member federation to register their Service Provider.
  • Harmonization of eduGAIN contacts by redirecting them to support@edugain.org

The Pilot tested the following assumptions on quantifying expected benefits:

Time saving on coordination work, specialist knowledge and communication overhead for research SPs and other similar SPs with a wide user footprint i.e. with users at IdPs in 5 or more countries.

  • In simplest case (est. 10% of cases), 20 minutes in contacting a range of IdPs, assuming all communication channelled by federation OR only one site per country impacted - c30 hours.
  • In majority case (est. 85%), 20 minutes in contacting a range of IdPs, assuming all communication channelled by federation OR only one site per country impacted – c50 hours.
  • In extreme cases (estimated 5%) save 4 days [4] - c.160 hours

Time saving on coordination work and communication overhead for federations where there is an incident impacting more than 2 entities:

  • In simplest case (est. 10% of cases), 5 mins per federation in contacting more than one other impacted party: c8.3 hours
  • In majority case (est. 85%), save 30 mins: c42 hours
  • In extreme cases (estimated 5%) save 1 day: c 40 hours
  • Total estimated saving per 100 cases that include > 2 impacted sites: just over 2 weeks, or EUR5000

The results of the pilot demonstrate that these assumptions were fair and that an eduGAIN Support Service would be beneficial to the user community.

5                      Summary

This CBA recommends entering into a production service of eduGAIN Support to provide greater support in interfederation performance issues.

Reasons for the recommendation are as follows:

  • The results of the pilot between March 2017 and November 2017 support that providing eduGAIN support is beneficial to the community
  • The available funding covers the costs in GN4-2 and GN4-3
  • Federations will benefit from not having to bear the full burden of interfederation communication
  • Federation role in support is preserved and reinforced by workflows that secure this
  • Supporting research communities is of strategic benefit to secure the future funding of eduGAIN
  • Supporting research communities with infrastructure is within the scope of the GÉANT strategy and the Trust and Identity strategy outlined by key NRENs in 2015 and beyond.

[1] This is based on the experience of the ORCID case of November 2016 where ORCID did not reach out for support for 5 days, but once they did, the case was resolved within 24 hours.

[2] ARIELY, D. (2010). Predictably irrational: the hidden forces that shape our decisions. New York, Harper Perennial .

[3] https://geant.app.box.com/s/8f30ptw5houmauurfqfupw3ruz3x9enu

[4] This is based on the experience of the ORCID case of November 2016 where ORCID did not reach out for support for 5 days, but once they did, the case was resolved within 24 hours.