Page tree

78

Cost Benefit Analysis

(CBA)

 

Purpose:

 

A cost-benefit analysis (CBA) is an analytical tool for assessing the advantages and disadvantages of moving forward with a business proposal or project. By using a cost-benefit analysis template, Activities can identify quantitative and monetary estimates to determine whether to pursue an initiative, tweak it or abandon it completely.

This document is your template for producing a GEANT cost benefit analysis. The CBA is created for the GEANT PMO by a project sponsor, department or unit seeking funding, approval, or both for an activity, initiative, or project.

 

Created by:

The CBA is created and maintained by the GEANT Activity Leader.

 

Submit to:

projectdocumentation@geant.org (the GÉANT Project Office).

 

CBA Information

Project Name:

Enhanced eduGAIN Support

CBA Author:

Ann Harding, Thomas Baerecke, Lukas Haemmerle, Justin Knight

CBA Code:

 

Project Type/Size

Medium

Date submitted

 

Activity (if applicable)

GN4-2 JRA3

Task(if applicable)

T2

Production Gate Approval Meeting

Decision:

 

Date of Meeting:

 

Comments:

 

 

Table of Contents

1 Executive Summary

1.1 Summary of work to date

1.2 Recommendation

1.3 Supporting Reasons

2 Phase 1. Background Information

2.1 Organisational Overview

3 Description Of Alternatives Considered

3.1 Option 1 – Do Nothing

3.2 Option 2 – Full First and Second Line support, with GÉANT SD

3.3 Option 3 – Manage response with distributed SLAs

5 Benefits and impacts….. ............................................................................................. .

6 Summary…..

1                      Executive Summary

The eduGAIN service interconnects identity federations around the world, simplifying access to content, services and resources for the global research and education community. Through eduGAIN, identity providers offer a greater range of services to their users as delivered by multiple federations in a truly collaborative environment; service providers offer their services to users in different federations thereby increasing their target market; and users seamlessly benefit from a wider range of services.

The eduGAIN interfederation service delivers a platform for the trustworthy exchange of metadata through the coordination of technical infrastructure and policy. This supports the needs of federations in establishing a common baseline for metadata interoperability and furthers the goals of federations to operate in a global identity access and service exchange.

Prior to the pilot of the enhanced eduGAIN Support Service, operational activities and support have been strictly limited to those related to the MDS - the central metadata service of eduGAIN, and supporting federations in correctly managing their metadata feeds to and from this system. The growing demand from research communities and from a set of federations to extend eduGAIN support beyond this scope lead to the development of the Enhanced eduGAIN Support Service pilot, to provide support and coordination for multi-party troubleshooting of performance affecting issues.

The pilot was approved under the GEANT PLM process on 1 st March 2017. This CBA analyses the results of the pilot, and in asking the question “is there a justifiable business case to proceed” recommends moving the Enhanced eduGAIN Support Service into production. Alongside this CBA, the PID, Service Design and Roadmap have also been updated in support of the recommended action.

 

1.1                Summary of work to date

Work began on Enhanced eduGAIN Support at the beginning of GN4-2, with the analysis of the then-current support channels for eduGAIN and gathering of intelligence around the requirement to enhance this. Under Task 2 of JRA3, a team of federation experts and supporting professionals designed a recommended service to take into pilot, which was approved by the GEANT PLM team on 1 st March 2017.

A part of the pilot, a dedicated ticket queue on the GEANT ORTS ticketing system was requested and configured, at zero cost (except some time) to the Task owing to GÉANT already having this availability. This ticketing queue was used to manage the support queries raised during the pilot and found fit-for-purpose. A rota of 2 people per week was established to manage queries, which was also found to be a suitable level of support availablility, with each person spending in the order of 2 hours per week addressing tickets raised.

The team not only responded to support requests and questions but also proactively helped reduce potential issues that could arise due to incorrect eduGAIN metadata. This was done by relying on external checking tools (mostly operated by UK federation) and by actively informing federation operators of the issues.

During the pilot, 78 tickets were processed spanning 23 different countries. Whilst fewer tickets were raised than anticipated, postive feedback was received by users of the pilot service (see 2.1.4 KPIs). The level of support requested against that which was anticipated does not indicate a lack of demand, but rather enables the analysis of specialities required and an update to the cost of time required to deliver a valuable service.

1.2                Recommendation

This Cost Benefit Analysis therefore recommends entering into a production service of Enhanced eduGAIN Support,   enhancing the staffing and scope of work of the central eduGAIN support team to support the coordination, troubleshooting, resolution and follow-up of complex multi-party performance or operational failures which involve multiple parties in eduGAIN.

A specific Service Design accompanies this CBA, and it is proposed the work closely with SA2 to operationalise the service.

1.3                Supporting Reasons

The reasons for this recommendation are that the results of the pilot in increasing in the size of the operational team of eduGAIN to support more complex troubleshooting delivers the following benefits:

  • Better user experience as issues are resolved more quickly and information shared more frequently and efficiently between affected parties.
  • More professional image for eduGAIN as a whole, especially among the e-Science users who do have direct knowledge of the eduGAIN brand, unlike more traditional users.
  • Time saving for federations in coordination overhead
  • Better knowledge and experience sharing with the wider community as a KB of experience will be created and made available .
  • Improved ability to engage collectively with vendors on bugs and feature issues as a greater picture of impact will be available
  • Increased value perception of eduGAIN to national and global identity federations, as well as research groups, driving adoption of eduGAIN and thereby greater service access globally
  • Improved metadata within eduGAIN, making issues less frequent and the overall eduGAIN service cleaner and more efficient to users
  • Assistance for SPs when unclear with which eduGAIN member federation to register their Service Provider.
  • Harmonization of eduGAIN contacts by redirecting them (e.g. edugain-integration@geant.net , edugain@geant.net ) to support@edugain.org

1.4 Summary Information

Over four years development costs are projected as €66,812 and the annual cost of production is estimated at €24,556, both fully funded by the GN projects. The benefit of the development is measured in terms of time and effort saved to the community, detailed below.

Time saving on coordination work, specialist knowledge and communication overhead for research SPs and other similar SPs with a wide user footprint i.e. with users at IdPs in 5 or more countries.

  • In simplest case (est. 10% of cases), 20 minutes in contacting a range of IdPs, assuming all communication channelled by federation OR only one site per country impacted - c30 hours.
  • In majority case (est. 85%), 20 minutes in contacting a range of IdPs, assuming all communication channelled by federation OR only one site per country impacted – c50 hours.
  • In extreme cases (estimated 5%) save 4 days [1] - c.160 hours

Time saving on coordination work and communication overhead for federations where there is an incident impacting more than 2 entities:

  • In simplest case (est. 10% of cases), 5 mins per federation in contacting more than one other impacted party: c8.3 hours
  • In majority case (est. 85%), save 30 mins: c42 hours
  • In extreme cases (estimated 5%) save 1 day: c 40 hours
  • Total estimated saving per 100 cases that include > 2 impacted sites: just over 2 weeks, or EUR5000

Recommended Approach – Summary information

Investment Information

Investment Costs

Total development cost over 4 years

66,812

 

Total production cost over 4 years

98,265

 

Total Investment Costs

 

165,078

 

GN4-2 JRA3 Funding

10,250

 

GN4-2 SA2 Funding

95,206

Forecast GN4-3 Funding

166,000

Total Funding

315,956

Technical Annex Reference (if applicable)

GN4-JRA3 T2 for development

GN4-2-SA2 T2 for operations

 

2                      Phase 1. Background Information

2.1                Organisational Overview

On a day-to day level, the eduGAIN interfederation service consists of two main elements:

        eduGAIN Policy Framework.

        Metadata Distribution Service (MDS).

The eduGAIN Policy Framework details the administrative and technical standards that all participant federations must adhere to in order to enable the trustworthy exchange of service information to support identity, authentication and authorisation between partner federations. It is administered by the eduGAIN Product Manager, who holds the role of moderator for the eduGAIN Steering Group (SG). The eduGAIN SG are representatives of the member federations of eduGAIN and are responsible for the technical governance of the service.

The key stakeholders for eduGAIN are they direct members who have specific roles and responsibilities in governing the technical service and carry out specific tasks in order to participate

  • National identity federations in GÉANT who are members of eduGAIN
  • Global identity federations

Prior to the Enhanced eduGAIN Support pilot, formal support activities of eduGAIN were targeted only at these groups.

The following additional groups benefit significantly from eduGAIN or provide benefit to eduGAIN but are a step removed, via identity federations.

  • Research communities
  • Campuses
  • Commercial services

For the majority of these groups, eduGAIN is, by design, an invisible service, an additional functionality offered by their national federation. The Enhanced eduGAIN Support pilot addressed how eduGAIN engages with them and in particular, a subset which are active primarily at an interfederation-only level and are aware of eduGAIN and contact eduGAIN addresses directly, despite pre-existing contracts with different federations and support channels available to them. Prior to the pilot, the only method of dealing with these mails was a polite note that we cannot help, and directing them to a database to look up IdP contacts individually, or to their respective federation who may do the same, or in some cases, more. This created a perceived lack of consistency and value in eduGAIN for these communities, despite the system overall working well when queries land in the most appropriate place. Through the Enhanced eduGAIN Support Service, these groups, as well as national and global identity federations have a single point of contact for eduGAIN support, with a team of experts working on rota providing both local and international expertise and co-ordination between affected parties.

2.1.1       Community Need

Prior to the Enhanced eduGAIN Support Service pilot, support queries for eduGAIN came in via varying channels. People contacted the eduGAIN operations team (OT), which normally handles technical issues concerning day-to-day operations. All other queries were forwarded somewhere else, such as the REFEDS (the Research and Education FEDerations group) mailing list, which might channel them back to someone else, including the eduGAIN OT. Via personal relationships within the larger eduGAIN community some people’s questions did arrive at their established contact, however this was at best informal and provided no means of measuring the level of support required in a centralised manner. Observing these inefficient and diffuse methods, the requirement for a uniform helpdesk as single point of contact for all eduGAIN-directed queries was identified. The Enhanced eduGAIN Support Service meets this requirement and serves both federations and SPs who have knowledge of eduGAIN.

Aggregating user questions in one single place also enables the federation operators to have a global, aggregated view of the number and scope of eduGAIN-specific questions coming in, making it easier to create a useful knowledge base and hence limiting the effort required to support eduGAIN users both centrally and directly at federations.

2.1.2       Drivers for Change

The research community, including CERN and LIGO have expressed significant doubts about the fitness of eduGAIN for their needs without a more active central operations team to support the more complex uses of federated identity that they require.

Federations are not all equally capable of bearing the burden of long-lasting and complex multi-party troubleshooting.

Federations that are able to carry this workload would still benefit by being freed up from coordination work to focus more closely on their directly involved users and customers.

Future funding for eduGAIN operations from the European Commission is likely to be strongly driven by the impact eduGAIN has on research and its use within research communities. In order to secure the funding that exists for the bare-bones version of eduGAIN Operations, it is therefore strategically important to invest in this area.

2.1.3       Outcomes

The outcomes of Enhanced eduGAIN support will be:

  • eduGAIN support enhanced to support performance-related incidents in interfederation
  • An aggregation of the many forms of contacting eduGAIN into a single ticketing system
  • Processes and workflows for coordinating performance incidents that have an interdomain scope
    • Processes will ensure the role of the federation operator is engaged and will not attempt to replace it centrally
  • A staffed rota of responders, delivered by federation experts
  • A KB/FAQ developed on an on-going basis and made available to the community
  • A full SD service during production, including response OLAs etc.
    • Delivered by a collaboration of GÉANT SD, GÉANT T&I experts and federation experts.

The Enhanced eduGAIN support team will be available to:

  • Federations who are troubleshooting on behalf of their IdPs or SPs
  • IdPs or SPs who have somehow found their way to eduGAIN independently with interfederation problems or queries

The main working tool for the Enhanced eduGAIN support team will be a ticketing system. The following requirements were identified for the system and an instance of GEANT’s OTRS queue was deployed, tested and found fit-for-purpose during the pilot:

  • Take account of the availability of 2nd level support persons when passing tickets from 1st level support. This may be implemented in two ways: the ticket might get assigned to 2nd level support as a whole group instead of an individual supporter. That way, a person currently on duty and available in 2nd level support can take the ticket up. The other option would be to keep first level support informed of the schedule for 2nd level support and eventually unplanned absences, so that they can assign tickets to one person knowing that the person is available and on duty.
  • There should be a possibility to create tickets via e-mail and via a web-form in order to make things easier for the requestors.
  • Federated login for admins and supporters would be nice to have.
  • Login as a guest should be possible with a restricted set of privileges.
  • There should be a ticket archive of resolved tickets which are kept at least for a limited period of time. This helps in coming back to the previous solutions for reoccurring issues, saving time in case the answer can be copied or adapted to the new case.
  • Adding (non-public) comments and notes to tickets helps to understand the ticket context and solution approach if similar cases arise in the future.
  • If specific federation operators are asked to assist for a specific case, they should be informed automatically of progress in the ticket if desired.

2.1.4       KPIS:

The CBA for the pilot phase specified the below KPIs:

KPI1: Target number of cases to be handled:

  • Pilot: 100 (Until Q3 2017)
  • Production Q4 2017: 50
  • Production 2018: 200
  • Production 2019: 300
  • Production 2020: 250

The number of tickets raised during the pilot was 78, coming from 23 different countries as detailed in the below table. Whilst less than anticipated, each ticket was resolved satisfactorially and the pilot service received very positive feedback from users (see table in KPI 3).

Month

Support tickets

Countries involved

April

1

Norway

May

2

Hungary, Switzerland

June

3

Germany, Belgium

July

9

Belgium, Estonia, USA, France, Denmark, Sweden, South Korea, Singapore, Uganda, Australia

August

5

Belgium, Spain, Iceland, Brazil, Canada

September

7

Spain, France, USA, Austria, Brazil, Israel, Slovenia

October

51

  US, Belgium, Czech Republic, Armenia, Lithuania, Slovenia, Luxembourg, Spain, Argentina, Portugal, Canada, Macdonia, Georgia, Australia, Ireland, Italy, Latvia, Sweden, Norway, France, Denmark, Greece,

 

The higher amount of tickets raised in October was as a result of a drive to clean up metadata in eduGAIN, and is not reflective of a typical month, however does highlight the importance of the pro-active work the support team undertook during the pilot.

For the production service, this CBA therefore recommends two KPIs for the number of cases handled; reactive tickets and pro-active tickets:

Reactive tickets :

  • Production 2018: 100
  • Production 2019: 150
  • Production 2020: 125

Proactive tickets (e.g. metadata clean-up, checking of error logs, identitfiying zombie IDs):

  • Production 2018: 25
  • Production 2019: 40
  • Production 2020: 30

KPI 2 : Federation satisfaction:

  • eduGAIN SG endorsement of enhancement at end of pilot.

The eduGAIN SG is in favour of the Enhanced eduGAIN Support Service going into production. [A1]

KPI 3: Research community satisfaction

  • Majority positive satisfaction ratings from research communities, by qualitative methods in pilot, use of feedback tools in production.

Postive feedback was received for a number of tickets from 11 countries, representing 10 federation operators and 3 SPs. Evidence of this was tracked through comments made and recorded in the ticketing system.

Function

Country

Comment

Federation Operator

France

Thanks a lot for your support, it's truly appreciated.

Federation Operator

Portugal

Thank you for the additional explanation.

Federation Operator

Argentina

Thank you very much for your advice and support.

Federation Operator

Czech Republic

Thanks for the report

Federation Operator

Slovenia

Thank you for your notice. It is also in our interest that these issues get fixed.

Federation Operator

Australia

Thanks for the notification.

Federation Operator

Ireland

Thanks [name of support contact], we'll fix that

Federation Operator

Norway

Thanks for the heads up.

Federation Operator

UK

Thanks for the update and excluding those IdPs from the check.

Federation Operator

Estonia

Thank you for your concern!

Existing SP

US

Thank you both for your quick replies.

Existing SP

UK

Many thanks for your reply, that’s really helpful.

Prospective SP

US

Thank you, this is very helpful.

3                      Description Of Alternatives Considered

3.1                Option 1 – Do Nothing

eduGAIN could continue with the current model and address these e-mails on an ad hoc basis, simply directing them to a different contact as appropriate.

While this would continue to keep the operational costs low, and addresses c2/3 of the use cases in eduGAIN (evaluated based on an analysis of the types of SP in eduGAIN), the remaining 1/3 consists of the complex research communities that the EC wishes e-Infrastructures to support. If eduGAIN is to make a case for continued funding of operations by the EC into GN4-3, these cases have to be visibly and enthusiastically supported, even if it were not already fundamentally compatible with the GÉANT strategy to do so.

In the event that the 'do nothing' option is chosen, GÉANT will need to look at alternative models of funding eduGAIN (e.g. by subscription) in the event that EC funding is tied to delivery of service to researchers. Even with this option, it is then likely that key research groups would collaborate to build an e-research specific alternative to eduGAIN as the technology aspects are simple to recreate, and users would then ultimately suffer by having to maintain multiple accounts.

 

3.2                Option 2 – Full First and Second Line support, with GÉANT SD

This is the option that has been piloted and is recommended to take forward to production. It combines the infrastructure and availability of service teams in GÉANT with the know-how of federation operators.

Details are in the section 2.1.3 on 'Outcomes'.

A sub-option of second level plus ticketing infrastructure only was evaluated in the pilot, however the tickets raised demonstrated a clear requirement for first and second level support.

3.3                Option 3 – Manage response with distributed SLAs

Harmonise the researcher experience of eduGAIN by adopting harmonised support with SLAs.

The experience with attempting to deliver multi domain services involving SLAs by NREN partners over the last 10 years has predominantly been unsuccessful. The ability of organisations to commit is not unified, and the change from social to contractual trust can result in loss of service, rather than an increase. [2]

At a federation level, not every federation has the depth of staff or scope of service to address complex e-Research needs. The 2016 REFEDS survey [3] (slide 5) indicates a wide gap between federations with significant investment and those with minimal manpower and budget. In addition, the primary business driver for almost every national identity federation is to serve the needs of the campus, rather than the service providers who are the most frequent contactors of eduGAIN centrally. This means that an approach where eduGAIN would recommend an SLA-type of arrangement via federations to serve these use cases would not be viable, even if it were acceptable to the community.

4                      Benefits / Impacts

Benfits of the Enhanced eduGAIN Support Service include:

  • Time saving for federations in coordination overhead
  • Better user experience as issues are resolved more quickly and information shared more frequently and efficiently between affected parties.
  • More professional image for eduGAIN as a whole, especially among the e-Science users who do have direct knowledge of the eduGAIN brand, unlike more traditional users.
  • Better knowledge and experience sharing with the wider community as a KB of experience will be created and made available .
  • Improved ability to engage collectively with vendors on bugs and feature issues as a greater picture of impact will be available.
  • Increased value perception of eduGAIN to national and global identity federations, as well as research groups, driving adoption of eduGAIN and thereby greater service access globally
  • Improved metadata within eduGAIN, making issues less frequent and the overall eduGAIN service cleaner and more efficient to users
  • Assistance for SPs when unclear with which eduGAIN member federation to register their Service Provider.
  • Harmonization of eduGAIN contacts by redirecting them (e.g. edugain-integration@geant.net , edugain@geant.net ) to support@edugain.org

The Pilot tested the following assumptions on quantifying expected benefits:

Time saving on coordination work, specialist knowledge and communication overhead for research SPs and other similar SPs with a wide user footprint i.e. with users at IdPs in 5 or more countries.

  • In simplest case (est. 10% of cases), 20 minutes in contacting a range of IdPs, assuming all communication channelled by federation OR only one site per country impacted - c30 hours.
  • In majority case (est. 85%), 20 minutes in contacting a range of IdPs, assuming all communication channelled by federation OR only one site per country impacted – c50 hours.
  • In extreme cases (estimated 5%) save 4 days [4] - c.160 hours

Time saving on coordination work and communication overhead for federations where there is an incident impacting more than 2 entities:

  • In simplest case (est. 10% of cases), 5 mins per federation in contacting more than one other impacted party: c8.3 hours
  • In majority case (est. 85%), save 30 mins: c42 hours
  • In extreme cases (estimated 5%) save 1 day: c 40 hours
  • Total estimated saving per 100 cases that include > 2 impacted sites: just over 2 weeks, or EUR5000

The results of the pilot demonstrate that these assumptions were fair and that an Enhanced eduGAIN Support Service would be beneficial to the user community.

5                      Summary

This CBA recommends entering into a production service of Enhanced eduGAIN Support (renaming it to just “eduGAIN Support”) to provide greater support in interfederation performance issues.

Reasons for the recommendation are as follows:

  • The results of the pilot between March 2017 and November 2017 support that providing enhanced eduGAIN support is beneficial to the community
  • The available funding covers the costs in GN4-2
  • The forecast funding beyond GN4-2 is within acceptable range of achieved GN4-2 funding for this kind of work
  • Federations will benefit from not having to bear the full burden of interfederation communication
  • Federation role in support is preserved and reinforced by workflows that secure this
  • Supporting research communities is of strategic benefit to secure the future funding of eduGAIN
  • Supporting research communities with infrastructure is within the scope of the GÉANT strategy and the Trust and Identity strategy outlined by key NRENs in 2015 and beyond.

[1] This is based on the experience of the ORCID case of November 2016 where ORCID did not reach out for support for 5 days, but once they did, the case was resolved within 24 hours.

[2] ARIELY, D. (2010). Predictably irrational: the hidden forces that shape our decisions. New York, Harper Perennial .

[3] https://geant.app.box.com/s/8f30ptw5houmauurfqfupw3ruz3x9enu

[4] This is based on the experience of the ORCID case of November 2016 where ORCID did not reach out for support for 5 days, but once they did, the case was resolved within 24 hours.


[A1]

The SG is being asked this, with comments until 12 th Dec.