Child pages
  • Meeting notes
Skip to end of metadata
Go to start of metadata

11 March 2013

Dear TCD pilot participants, all,

This is my regular e-mail update on the progress of the TERENA Trusted
Cloud Drive pilot. You can always take a look at the latest updates at
https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN

My last update was in December 2012, my apologies for the long radio
silence. The actual figures of the pilot are as follows:
- The total number of platform test/dev installations has raised to 9
(+2): TERENA, PSNC, BELNET, CESNET, GRNET, FCCN, Srce/CARNet, RNP, CESCA
- 46 user accounts (+0) are enabled on the TERENA pilot service at
https://tc2.terena.org
- The cloud-pilot mailing list has 62 members (+1).

The small increase in these figures during the last 3 months means that
the interest generated by the pilot within the TERENA community has
reached its maximum and stabilised around those numbers.

*Major milestones*

As it was reported at the last TF-Storage meeting in Berlin (take a look
at my slides
http://www.terena.org/activities/tf-storage/ws14/slides/2013036-TCDpilot.pdf)
the pilot has reached several major milestones:

1) CESNET (Jakub Peisar) has done the performance/scalability tests of
the TCD platform, the results and conclusions can be found at
https://confluence.terena.org/display/CloudStorage/Software+tests

2) GRNET (Christos Loverdos) has implemented the TCD integration with
Pithos+ back-end. The code has been contributed to the TERENA Github and
the work was successfully demonstrated to TF-Storage in Berlin
http://www.terena.org/activities/tf-storage/ws14/slides/20130306-Pithos-CloudDrive.pdf

3) RNP (Gui Maluf) has implemented the OpenStack Object Storage (Swift)
S3 API to the TCD platform. The code has been contributed to the TERENA
Github https://github.com/TERENA/CloudDrive
It is planned to integrate TCD (via the OpenStack S3 API) to
RNP's cloud storage infrastructure.

4) CESCA (Lorenzo J. Cubero) is progressing with the Jclouds API
implementation to the TCD platform. Progress will soon be reported.

These software developments and platform integration efforts allow us to
focus on the service development and validation aspects of the pilot in
its last month (until April 2013)!

On behalf of the community, TERENA very much appreciates the work
committed by these people and their organisations.

*TCD platform directions*

Note that Maarten Koopmans (Vrijheid.net), the lead developer of the
platform, informed TERENA that he's going to consolidate the code base;
better Scala, less dependencies, smaller code size are expected. The
main Github branch will be closed however the TERENA Gitghub fork
remains open.

Concerning the features and functions of the TCD platform the following
strategic direction has been envisaged by TERENA:
- TCD should focus on its main asset; maintaining trust and privacy by
separating metadata and encryption keys (kept in a trusted place) from
the storage data (exported to back-end storage).
- TCD should not compete with the feature-rich front-end applications
available on the market but broker them to various storage back-ends.
- Therefore, TCD should be a thin layer (controlled and operated by
NRENs) separating/interfacing end-user application domains and cloud
service provider domains; considered to be a "storage middleware".
- The integration of TCD should be considered with both end-user
oriented storage applications and back-end storage cloud provider
platforms.

*Use cases and service validation scenarios*

In line with the strategic direction outlined above, the pilot will
further explore:

A) the service brokering scenarios made available by the TCD-Pithos+ and
the TCD-Swift integrations; different NREN storage infrastructures
and/or public cloud storage back-ends can be brokered down to user
domains in a trusted, privacy-controlled way.

B) the service integration scenarios with feature-rich end-user storage
applications; both community-developed and commercial solutions (such as
OwnCloud and PowerFolder) will be investigated for potential integration
with TCD functionality to preserve trust and privacy.

That's it for now! I'd like to encourage everyone to contribute to the
service development and validation aspects of the pilot in its very last
month.

 

17 December 2012

Dear TCD pilot participants, all,

This is my regular e-mail update on the progress of the TERENA Trusted
Cloud Drive pilot (the last one in 2012) however, you can always take a
look at the latest updates at
https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN

The last pilot conf.call was on 14 November 2012, the minutes can be
found at
https://confluence.terena.org/display/CloudStorage/Meeting+notes

Since then, FCCN (Portugal), CESNET (Czech Rep.) and RNP (Brazil)
managed to install a service platform instance for experimental purposes
at their locations. The total number of platform test installations has
raised to 7. 43 user accounts are enabled on the TERENA pilot service
https://tc2.terena.org. The cloud-pilot mailing list has 62 members.

Thanks to CESNET, the Installation Guide - maintained by the community -
has been significantly improved with useful fixes, tips and tricks.
Please, take a look at
https://confluence.terena.org/display/CloudStorage/Installation+Guide

The various software tests (performance, security, etc) are on-going at
CESNET and PSNC. Feel free to offer any software testing effort to the
pilot.

The voluntary software developers have been working on the platform
compatibility improvements, namely:
- Lorenzo (CESCA) on the Jclouds back-end API
- Gui (RNP) on the OpenStack Swift integration
- Christos (GRNET) on the Pithos+ interworking
I would like to *encourage everyone* to post questions, updates, useful
hints on the mailing list of the pilot.

TERENA has created an Github account (as virtual organisation)
https://github.com/terena and forked CloudDrive as a project. Developers
can be added as members of the organisation administered by TERENA. The
main CloudDrive branch is still maintained by Maarten and we try to keep
up the consistency as the development efforts are converging.

There are plans to initiate a dialogue (and potential partnership) with
the Helix Nebula http://helix-nebula.eu/ project as well as the
forthcoming GN3+ project could related (SA7) activity. This is still in
embryonic phase, we have to see the potentials and the level of uptake
of TCD in the community.

TERENA has recently received some interest from commercials willing to
provide various cloud services to the R&E community. TERENA TF-Storage
group had a discussion with Joyent
https://www.terena.org/events/details.php?event_id=2473
where TCD was mentioned as a potential service platform brokering
Joyent's IaaS offerings to the community. TERENA will continue the
discussion with Joyent in early 2013.

Keep up the good work in 2013!

Season's greetings,
Peter

14 November 2012

Participants:
Panos Louridas, GRNET
Christos KK Loverdos, GRNET
Lorenzo J. Cubero, CESCA
Gui Maluf, RNP
Peter Szegedi, TERENA


NOTES
Previously (via e-mail)
1) Lorenzo (CESCA) has offered to work on the Jclouds API implementation
and testing at the platform back-end. He has been talking to Adrian Cole
[jcloud founder] recently.

http://www.jclouds.org/documentation/reference/supported-providers/

2) Gui (RNP) has offered to make a bridge between OpenStack Swift and TERENA cloud broker. AFAIK, RNP has an OpenStack based cloud storage infrastructure. http://docs.openstack.org/trunk/openstack-object-storage/admin/content/configuring-openstack-object-storage-with-s3_api.html

3) Christos (GRNET) has just offered to make a bridge between Pithos+ (OpenStack based) and the TERENA cloud broker. He has been working with GRNET on Okeanos - their overall cloud storage service - and more specifically on its accounting components, written in Scala. He is an expert in Scala.

Lorenzo has started to work on the Jclouds API implementation. He also managed
to install the platform at his computer using the VMware images available. He is
now trying to install it right from Github, manually. Gui has failed with the first
attempt, and he is seeking for help on the mailing list. Christos will try and install
the platform at GRNET, too.

The very first Milestone to Lorenzo, Gui and Christos would be to get the development platform up and running at their location.

Panos asked about the code license and the Github account. Peter said that the license is
Apache Licenses, Version 2.0. The Github account is owned by the lead developer (Maarten Koopmans)
at the moment.

Action on Peter (TERENA) to talk to Maarten about the migration of the Github account to TERENA.

Participants agreed to coordinate efforts and work together towards the common goal. TERENA will
facilitate this development work by providing the mailing list, common wiki space, and access to
the Github account.
Next development meeting will be called as soon as the first Milestone has been reached.

 


26 October 2012

PARTICIPANTS:
Jean-Philippe Evrard, Belnet
Jakub Peisar, CESNET
Maarten Koopmans
Szabolcs Szekelyi, NIIF
Maciej Brzezniak, PSNC
Lorenzo J. Cubero, CESCA
Peter Szegedi, TERENA
Joël Casutt, SWITCH
Roberto Araujo, RNP
Panos Louridas, GRNET
If I missed you and you want to be minuted, let me know! NOTES: 1) *Pilot status update* Peter (TERENA): The source code, its documentation, the Installation Guide made by PSNC, and the downloadable VMware images (made by Maarten) are all available on the Wiki https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN The pilot is running until March 2013 (five months left). The aims of the pilot are: a) to ensure the sustainability of the service platform (i.e. gather the developers' community around the code that can implement the roadmap (new features, bug fixes) agreed). b) to ensure the required scalability, performance, and stability of the code (i.e. run coordinated tests on several implementations of the platform and come up with recommendations for improvements). c) to investigate the potential service delivery models using the cloud storage broker platform (i.e. collect use cases, make a list of necessary features to be implemented, and agree on a common roadmap). As of today, TERENA and PSNC have already installed 1-1 service instances, and BELNET, CESNET, Scre, FCCN and CESCA signaled that they also plan to install the platform at their location. This is considered as the core group. There are >25 test accounts provisioned to the TERENA installation so far. This group is considered as the core of alpha/beta testers form a user perspective. 2) *Software tests* Maarten: Previously, the platform has been tested on EC2 resources. Three virtual servers have been used as a redundant Metadata store and two additional virtual servers for WebDAV and Website. This is the recommended setup for testing. Maciej (PSNC): PSNC will start with the security tests first and do a code review. Later on performance and scalability tests (e.g., stresstest) will also be done. In case of interest, PSNC plan to test various user applications on top of WebDAV (maybe next year). Jean-Philippe (Belnet): The platform has been installed on one single server at BELNET. Once the on-going storage infrastructure tendering is over, Belnet plans to do some test too. The software tests must be coordinated. Jakub (CESNET): CESNET will also install the platform on several virtual servers as suggested. Performance test (starting with the Metadata store) will then be done in a well-documented way. *ACTION* 1 on Peter (TERENA) to coordinate and compile a list of test procedures and call for potential testers on the mailing list. Maarten commented that the file level compression and encryption is configurable (can be turned off) that can immediately improve the performance with a factor, if needed. The Voldemort metadata store is also linearly scaling, additional virtual servers can be added as the metadata increases. 3) *Software development* Maarten: Replication/Synchronisation between two or more data storage back-ends is not implemented at the moment and it is not the primary aim. Szabolcs (NIIF): It is easier to implement replication at the back-end storage level rather than in the broker platform on top. It is recommended NOT to deal with the replication at the platform level. Maarten: The implementation of "sharing" was in the original development roadmap, but Maarten is fully booked by the end of the year with other projects so, no progress is expected from their side. Peter: An other necessary feature would be an administrative web interface. That can help all the service adopters to configure the platform easily. This development can be done by a web developer, if agreed. Peter (TERENA): In principle, the service development procedure would be that the pilot participants come up with an agreed list of features to be implemented, and then the actual coders (Maarten or others) will be selected once the roadmap is clear. Lorenzo (CESCA): Lorenzo volunteered to contribute to the pilot and implement a Jclouds back-end API as well as do some performance tests on it. The idea behind Jclouds API is to ensure the full flexibility of the platform in terms of storage back-ends. In theory, Jclouds makes it possible to configure the actual storage API used. *ACTION* 2 on Lorenzo (CESCA) to start working on the Jclouds implementation ASAP and report back to the pilot participants by the end of the year. Lorenzo also asked if a flexible front-end API (e.g., REST API) would be of interest. Peter answered that the priority would be to ensure the largest back-end API compatibility possible. NRENs are looking for solutions to be put on top of their storage infrastructure (using different APIs) and provide services to their users quick and easy. The meeting was closed. No agreement on the next call, to be clarified on the mailing list. LIST OF ACTIONS: ACTION 1 on Peter (TERENA) to coordinate and compile a list of test procedures and call for potential testers on the mailing list. ACTION 2 on Lorenzo (CESCA) to start working on the Jclouds implementation ASAP and report back to the pilot participants by the end of the year.


03 October 2012

Dear all,

This is my regular update on the progress of the TERENA Trusted Cloud
Drive pilot. More about the pilot can be found at
http://terena.org/clouddrive

We had an excellent TF-Storage task force meeting in Dubrovnik last week
where we discussed about the Trusted Cloud Drive, obviously. I wanted to
find answers to my questions posted before, namely:

1) Who else is interested in installing the cloud platform at their
location for experimental purposes?
At the meeting, BELNET, CESNET and Scre/CARNet mentioned that they are
willing to install an instance of the platform at their location for
experimental purposes. In addition to this, NIIF mentioned that they are
investigating if one of the Hungarian universities wants to install the
platform (just like FileSender). I have added a small table to the Wiki
where we can collect the installations known
https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN

Please, get back to me if there's other interest out there?

2) Who is interested in introducing this cloud platform as a service
and what would be the main characteristics of such a service?
This question was discussed in light of the previous one, and the
aforementioned NRENs are obviously willing to investigate the potential
service development options. To this end, I extended the Service
Delivery Scenarios page with two additional tables
https://confluence.terena.org/display/CloudStorage/Service+Delivery+Scenarios

The first one shows the potential trust models mainly inspired by the
very interesting Cloud Legal Project talk
http://www.cloudlegal.ccls.qmul.ac.uk//Research/
Assuming a multi-layer user-uni-nren-terena-commercial model, the host
of the Cloud Broker holds the TRUST with the end-user and of course the
Clod Broker platform can move up and down in the stack depending on the
service/trust model preferred. Commercial cloud storage solutions are
shown in the last column for comparison.

The second table depicts a concept of a TERENA Storage franchise model.
The idea is that universities or other institutes (who wants to provide
a cloud storage service for end-users) can download the Cloud Drive
platform (for free) and the desired storage back-end API plugins (under
specific API and SLA agreements) from a TEREAN-hosted portal. NRENs and
commercials can submit their storage offers, necessary plugins and
conditions to the portal. TERENA can act as a clearing house. This idea
must be further polished, of course.

3) Which API development option would you prefer, purely from your
storage back-end and service development point of view?
This discussion was postponed to a later phase. We have to understand
the desired development roadmap of the Cloud Drive platform first and
see the clear directions that we should take.

To support the potential platform installations, PSNC has made available
their Installation Guide at
https://confluence.terena.org/display/CloudStorage/Installation+Guide
and Maarten will provide downloadable VMware images for Linux for quick
start.

TERENA on behalf of PSNC has applied for an Amazon research grant. If
that application is supported we hopefully can give away some Amazon S3
credits to those who wants to experiment with the Trusted Cloud Drive
platform with an S3 back-end at their organisation.

More is coming soon....

12 September 2012

Dear all,

Let me give you a brief update on what's boiling in the pot at the
moment regarding the TERENA Trusted Cloud Drive pilot
https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN
We'll have a chance to talk about these at TF-Storage or at any time on
the mailing list.

First of all, congratulations to PSNC for the *first installation* of
the cloud platform at an NREN location!

PSNC is willing to share their practical experiences with the entire
installation process (in addition to the documentation found on the
Wiki), as well as they offer pre-installation support to any new
releases of the software, if appropriate. This leads us to the first
question:

1) Who else is interested in installing the cloud platform at their
location for experimental purposes?

It is planned to give technical assistance to the local installations,
if needed. PSNC's documentation will be available on the Wiki, Maarten
(the lead developer) will be able to give limited support and make the
development image available for download, as well as new software
releases will come out. Do note that at the moment Maarten is working of
the tool updates according to his own agenda, therefore it would be
beneficial to know what our common roadmap for the tool would be and
steer the developments into that direction. So, the second question is:

2) Who is interested in introducing this cloud platform as a service and
what would be the main characteristics of such a service?

Obviously, the potential service models are subject to discussion. You
can find some high level scenarios at
https://confluence.terena.org/display/CloudStorage/Service+Delivery+Scenarios
Whether NRENs are willing to offer their own storage capacity as
back-end or we look for commercial cloud storage offerings, the storage
API development is a key issue. Three potential options are envisaged at
the moment:
a) Develop a direct bridge to your storage back-end.
b) Develop a generic S3 API (independent from Amazon)
c) Develop a Jclouds API (to comply with many other interfaces)
We have to know first who is interested in hooking up the cloud platform
with their in-house storage and what those cloud storage implementations
are based on. To this end, we've come to the third question:

3) Which API development option would you prefer, purely from your
storage back-end and service development point of view?

Please, think about these three questions and get back to me, to the
list, or to the TF-Stogare meeting and discuss!!!
http://www.terena.org/activities/tf-storage/ws13/

 


06 August 2012

Dear pilot participants,

The summer period is undoubtedly a bit silent but those who are not on
vacation can play with the TERENA installation of the Trusted Cloud
Drive platform and collect some hands-on experience, I hope.

At the moment, 14 people have access to the service and I've already
received some feedback from the beta testers. I'd just like to encourage
all of you to get access, play with the tool, and give us some feedback.
https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN

Starting from September, we can speed up the work and start looking at
the potential service delivery scenarios
https://confluence.terena.org/display/CloudStorage/Service+Delivery+Scenarios
More importantly, the pilot participants with firm interest and
dedicated efforts can join the development team and become the co-owner
of the code that ensures the long-term sustainability.

TF-Storage organises a meeting on 26-27 September 2012, in Dubrovnik,
Croatia.
http://www.terena.org/activities/tf-storage/ws13/
Part of the meeting will be dedicated to the cloud pilot discussion,
initial feedback, and pilot roadmap. I'd like to invite everyone to
register to the meeting and participle ether in person or remotely
https://www.terena.org/events/details.php?event_id=2317

 


26 June 2012

Dear all,

First of all, thank you very much for your interest in the TERENA
Trusted Cloud Drive pilot! At the moment we have 24 people on this
'cloud-pilot' mailing list. You can check the current list of
participants at
https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN

This list is strictly for the active pilot participants contributing to
Phase ii. If you just want to be informed about the major milestones of
the pilot you can join the 'storage' or the 'nren-clouds' lists of
TERENA http://www.terena.org/mailinglists.php

I have updated the Wiki page with two things:

1) How to get access to the TERENA installation of the service for
initial testing
https://confluence.terena.org/display/CloudStorage/Get+access+to+the+TERENA+installation

Please, follow the instructions on the Wiki and e-mail me your id to be
whitelisted by the end of this week. Do note that due to the current
limitations of the setup the number of test users is limited!

2) Preliminary service delivery scenarios
https://confluence.terena.org/display/CloudStorage/Service+Delivery+Scenarios

One of the aims of the pilot is to design, implement, test, and validate
various potential service delivery scenarios. I depicted four basic
scenarios on the Wiki. The first scenario is basically the one
implemented at TERENA. Other potential scenarios are subject to discussion.


18 June 2012

Dear participants,

This is my very first mail to the list. I maintain this list to collect
the participants (technical contacts) interested in participating the
public trial phase of the TERENA Trusted Cloud Drive pilot.

If you just want to be informed about the major steps and progress of
the pilot, you don't have to be on this list! I'll inform the TF-Storage
task force list as well as the NREN-cloud list about the milestones of
the pilot so that you can follow those lists. This list is mainly for
the active participants of the pilot!

Please, take a look at the Wiki page of Phase ii.
https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN
I've added the list of interested participants there, as of today. Let
me know if you need any update there!

I'm still waiting for some confirmations before the list is more of less
complete and I can call for the kick-off VC. Stay tuned!
  • No labels