11 March 2013
Dear TCD pilot participants, all, This is my regular e-mail update on the progress of the TERENA Trusted Cloud Drive pilot. You can always take a look at the latest updates at https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN My last update was in December 2012, my apologies for the long radio silence. The actual figures of the pilot are as follows: - The total number of platform test/dev installations has raised to 9 (+2): TERENA, PSNC, BELNET, CESNET, GRNET, FCCN, Srce/CARNet, RNP, CESCA - 46 user accounts (+0) are enabled on the TERENA pilot service at https://tc2.terena.org - The cloud-pilot mailing list has 62 members (+1). The small increase in these figures during the last 3 months means that the interest generated by the pilot within the TERENA community has reached its maximum and stabilised around those numbers. *Major milestones* As it was reported at the last TF-Storage meeting in Berlin (take a look at my slides http://www.terena.org/activities/tf-storage/ws14/slides/2013036-TCDpilot.pdf) the pilot has reached several major milestones: 1) CESNET (Jakub Peisar) has done the performance/scalability tests of the TCD platform, the results and conclusions can be found at https://confluence.terena.org/display/CloudStorage/Software+tests 2) GRNET (Christos Loverdos) has implemented the TCD integration with Pithos+ back-end. The code has been contributed to the TERENA Github and the work was successfully demonstrated to TF-Storage in Berlin http://www.terena.org/activities/tf-storage/ws14/slides/20130306-Pithos-CloudDrive.pdf 3) RNP (Gui Maluf) has implemented the OpenStack Object Storage (Swift) S3 API to the TCD platform. The code has been contributed to the TERENA Github https://github.com/TERENA/CloudDrive It is planned to integrate TCD (via the OpenStack S3 API) to RNP's cloud storage infrastructure. 4) CESCA (Lorenzo J. Cubero) is progressing with the Jclouds API implementation to the TCD platform. Progress will soon be reported. These software developments and platform integration efforts allow us to focus on the service development and validation aspects of the pilot in its last month (until April 2013)! On behalf of the community, TERENA very much appreciates the work committed by these people and their organisations. *TCD platform directions* Note that Maarten Koopmans (Vrijheid.net), the lead developer of the platform, informed TERENA that he's going to consolidate the code base; better Scala, less dependencies, smaller code size are expected. The main Github branch will be closed however the TERENA Gitghub fork remains open. Concerning the features and functions of the TCD platform the following strategic direction has been envisaged by TERENA: - TCD should focus on its main asset; maintaining trust and privacy by separating metadata and encryption keys (kept in a trusted place) from the storage data (exported to back-end storage). - TCD should not compete with the feature-rich front-end applications available on the market but broker them to various storage back-ends. - Therefore, TCD should be a thin layer (controlled and operated by NRENs) separating/interfacing end-user application domains and cloud service provider domains; considered to be a "storage middleware". - The integration of TCD should be considered with both end-user oriented storage applications and back-end storage cloud provider platforms. *Use cases and service validation scenarios* In line with the strategic direction outlined above, the pilot will further explore: A) the service brokering scenarios made available by the TCD-Pithos+ and the TCD-Swift integrations; different NREN storage infrastructures and/or public cloud storage back-ends can be brokered down to user domains in a trusted, privacy-controlled way. B) the service integration scenarios with feature-rich end-user storage applications; both community-developed and commercial solutions (such as OwnCloud and PowerFolder) will be investigated for potential integration with TCD functionality to preserve trust and privacy. That's it for now! I'd like to encourage everyone to contribute to the service development and validation aspects of the pilot in its very last month.
17 December 2012
Dear TCD pilot participants, all, This is my regular e-mail update on the progress of the TERENA Trusted Cloud Drive pilot (the last one in 2012) however, you can always take a look at the latest updates at https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN The last pilot conf.call was on 14 November 2012, the minutes can be found at https://confluence.terena.org/display/CloudStorage/Meeting+notes Since then, FCCN (Portugal), CESNET (Czech Rep.) and RNP (Brazil) managed to install a service platform instance for experimental purposes at their locations. The total number of platform test installations has raised to 7. 43 user accounts are enabled on the TERENA pilot service https://tc2.terena.org. The cloud-pilot mailing list has 62 members. Thanks to CESNET, the Installation Guide - maintained by the community - has been significantly improved with useful fixes, tips and tricks. Please, take a look at https://confluence.terena.org/display/CloudStorage/Installation+Guide The various software tests (performance, security, etc) are on-going at CESNET and PSNC. Feel free to offer any software testing effort to the pilot. The voluntary software developers have been working on the platform compatibility improvements, namely: - Lorenzo (CESCA) on the Jclouds back-end API - Gui (RNP) on the OpenStack Swift integration - Christos (GRNET) on the Pithos+ interworking I would like to *encourage everyone* to post questions, updates, useful hints on the mailing list of the pilot. TERENA has created an Github account (as virtual organisation) https://github.com/terena and forked CloudDrive as a project. Developers can be added as members of the organisation administered by TERENA. The main CloudDrive branch is still maintained by Maarten and we try to keep up the consistency as the development efforts are converging. There are plans to initiate a dialogue (and potential partnership) with the Helix Nebula http://helix-nebula.eu/ project as well as the forthcoming GN3+ project could related (SA7) activity. This is still in embryonic phase, we have to see the potentials and the level of uptake of TCD in the community. TERENA has recently received some interest from commercials willing to provide various cloud services to the R&E community. TERENA TF-Storage group had a discussion with Joyent https://www.terena.org/events/details.php?event_id=2473 where TCD was mentioned as a potential service platform brokering Joyent's IaaS offerings to the community. TERENA will continue the discussion with Joyent in early 2013. Keep up the good work in 2013! Season's greetings, Peter
14 November 2012
Participants:
Panos Louridas, GRNET
Christos KK Loverdos, GRNET
Lorenzo J. Cubero, CESCA
Gui Maluf, RNP
Peter Szegedi, TERENA
NOTES
Previously (via e-mail)
1) Lorenzo (CESCA) has offered to work on the Jclouds API implementation and testing at the platform back-end. He has been talking to Adrian Cole [jcloud founder] recently. http://www.jclouds.org/documentation/reference/supported-providers/
2) Gui (RNP) has offered to make a bridge between OpenStack Swift and TERENA cloud broker. AFAIK, RNP has an OpenStack based cloud storage infrastructure. http://docs.openstack.org/trunk/openstack-object-storage/admin/content/configuring-openstack-object-storage-with-s3_api.html
3) Christos (GRNET) has just offered to make a bridge between Pithos+ (OpenStack based) and the TERENA cloud broker. He has been working with GRNET on Okeanos - their overall cloud storage service - and more specifically on its accounting components, written in Scala. He is an expert in Scala.
Lorenzo has started to work on the Jclouds API implementation. He also managed
to install the platform at his computer using the VMware images available. He is
now trying to install it right from Github, manually. Gui has failed with the first
attempt, and he is seeking for help on the mailing list. Christos will try and install
the platform at GRNET, too.
The very first Milestone to Lorenzo, Gui and Christos would be to get the development platform up and running at their location.
Panos asked about the code license and the Github account. Peter said that the license is
Apache Licenses, Version 2.0. The Github account is owned by the lead developer (Maarten Koopmans)
at the moment.
Action on Peter (TERENA) to talk to Maarten about the migration of the Github account to TERENA.
Participants agreed to coordinate efforts and work together towards the common goal. TERENA will
facilitate this development work by providing the mailing list, common wiki space, and access to
the Github account.
Next development meeting will be called as soon as the first Milestone has been reached.
26 October 2012
PARTICIPANTS: Jean-Philippe Evrard, Belnet Jakub Peisar, CESNET Maarten Koopmans Szabolcs Szekelyi, NIIF Maciej Brzezniak, PSNC Lorenzo J. Cubero, CESCA Peter Szegedi, TERENA Joël Casutt, SWITCH
Roberto Araujo, RNP
Panos Louridas, GRNET
If I missed you and you want to be minuted, let me know! NOTES: 1) *Pilot status update* Peter (TERENA): The source code, its documentation, the Installation Guide made by PSNC, and the downloadable VMware images (made by Maarten) are all available on the Wiki https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN The pilot is running until March 2013 (five months left). The aims of the pilot are: a) to ensure the sustainability of the service platform (i.e. gather the developers' community around the code that can implement the roadmap (new features, bug fixes) agreed). b) to ensure the required scalability, performance, and stability of the code (i.e. run coordinated tests on several implementations of the platform and come up with recommendations for improvements). c) to investigate the potential service delivery models using the cloud storage broker platform (i.e. collect use cases, make a list of necessary features to be implemented, and agree on a common roadmap). As of today, TERENA and PSNC have already installed 1-1 service instances, and BELNET, CESNET, Scre, FCCN and CESCA signaled that they also plan to install the platform at their location. This is considered as the core group. There are >25 test accounts provisioned to the TERENA installation so far. This group is considered as the core of alpha/beta testers form a user perspective. 2) *Software tests* Maarten: Previously, the platform has been tested on EC2 resources. Three virtual servers have been used as a redundant Metadata store and two additional virtual servers for WebDAV and Website. This is the recommended setup for testing. Maciej (PSNC): PSNC will start with the security tests first and do a code review. Later on performance and scalability tests (e.g., stresstest) will also be done. In case of interest, PSNC plan to test various user applications on top of WebDAV (maybe next year). Jean-Philippe (Belnet): The platform has been installed on one single server at BELNET. Once the on-going storage infrastructure tendering is over, Belnet plans to do some test too. The software tests must be coordinated. Jakub (CESNET): CESNET will also install the platform on several virtual servers as suggested. Performance test (starting with the Metadata store) will then be done in a well-documented way. *ACTION* 1 on Peter (TERENA) to coordinate and compile a list of test procedures and call for potential testers on the mailing list. Maarten commented that the file level compression and encryption is configurable (can be turned off) that can immediately improve the performance with a factor, if needed. The Voldemort metadata store is also linearly scaling, additional virtual servers can be added as the metadata increases. 3) *Software development* Maarten: Replication/Synchronisation between two or more data storage back-ends is not implemented at the moment and it is not the primary aim. Szabolcs (NIIF): It is easier to implement replication at the back-end storage level rather than in the broker platform on top. It is recommended NOT to deal with the replication at the platform level. Maarten: The implementation of "sharing" was in the original development roadmap, but Maarten is fully booked by the end of the year with other projects so, no progress is expected from their side. Peter: An other necessary feature would be an administrative web interface. That can help all the service adopters to configure the platform easily. This development can be done by a web developer, if agreed. Peter (TERENA): In principle, the service development procedure would be that the pilot participants come up with an agreed list of features to be implemented, and then the actual coders (Maarten or others) will be selected once the roadmap is clear. Lorenzo (CESCA): Lorenzo volunteered to contribute to the pilot and implement a Jclouds back-end API as well as do some performance tests on it. The idea behind Jclouds API is to ensure the full flexibility of the platform in terms of storage back-ends. In theory, Jclouds makes it possible to configure the actual storage API used. *ACTION* 2 on Lorenzo (CESCA) to start working on the Jclouds implementation ASAP and report back to the pilot participants by the end of the year. Lorenzo also asked if a flexible front-end API (e.g., REST API) would be of interest. Peter answered that the priority would be to ensure the largest back-end API compatibility possible. NRENs are looking for solutions to be put on top of their storage infrastructure (using different APIs) and provide services to their users quick and easy. The meeting was closed. No agreement on the next call, to be clarified on the mailing list. LIST OF ACTIONS: ACTION 1 on Peter (TERENA) to coordinate and compile a list of test procedures and call for potential testers on the mailing list. ACTION 2 on Lorenzo (CESCA) to start working on the Jclouds implementation ASAP and report back to the pilot participants by the end of the year.
03 October 2012
Dear all, This is my regular update on the progress of the TERENA Trusted Cloud Drive pilot. More about the pilot can be found at http://terena.org/clouddrive We had an excellent TF-Storage task force meeting in Dubrovnik last week where we discussed about the Trusted Cloud Drive, obviously. I wanted to find answers to my questions posted before, namely:
1) Who else is interested in installing the cloud platform at their location for experimental purposes?
At the meeting, BELNET, CESNET and Scre/CARNet mentioned that they are willing to install an instance of the platform at their location for experimental purposes. In addition to this, NIIF mentioned that they are investigating if one of the Hungarian universities wants to install the platform (just like FileSender). I have added a small table to the Wiki where we can collect the installations known https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN Please, get back to me if there's other interest out there?
2) Who is interested in introducing this cloud platform as a service and what would be the main characteristics of such a service?
This question was discussed in light of the previous one, and the aforementioned NRENs are obviously willing to investigate the potential service development options. To this end, I extended the Service Delivery Scenarios page with two additional tables https://confluence.terena.org/display/CloudStorage/Service+Delivery+Scenarios The first one shows the potential trust models mainly inspired by the very interesting Cloud Legal Project talk http://www.cloudlegal.ccls.qmul.ac.uk//Research/ Assuming a multi-layer user-uni-nren-terena-commercial model, the host of the Cloud Broker holds the TRUST with the end-user and of course the Clod Broker platform can move up and down in the stack depending on the service/trust model preferred. Commercial cloud storage solutions are shown in the last column for comparison. The second table depicts a concept of a TERENA Storage franchise model. The idea is that universities or other institutes (who wants to provide a cloud storage service for end-users) can download the Cloud Drive platform (for free) and the desired storage back-end API plugins (under specific API and SLA agreements) from a TEREAN-hosted portal. NRENs and commercials can submit their storage offers, necessary plugins and conditions to the portal. TERENA can act as a clearing house. This idea must be further polished, of course.
3) Which API development option would you prefer, purely from your storage back-end and service development point of view?
This discussion was postponed to a later phase. We have to understand the desired development roadmap of the Cloud Drive platform first and see the clear directions that we should take. To support the potential platform installations, PSNC has made available their Installation Guide at https://confluence.terena.org/display/CloudStorage/Installation+Guide and Maarten will provide downloadable VMware images for Linux for quick start. TERENA on behalf of PSNC has applied for an Amazon research grant. If that application is supported we hopefully can give away some Amazon S3 credits to those who wants to experiment with the Trusted Cloud Drive platform with an S3 back-end at their organisation. More is coming soon....
12 September 2012
Dear all, Let me give you a brief update on what's boiling in the pot at the moment regarding the TERENA Trusted Cloud Drive pilot https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN We'll have a chance to talk about these at TF-Storage or at any time on the mailing list. First of all, congratulations to PSNC for the *first installation* of the cloud platform at an NREN location! PSNC is willing to share their practical experiences with the entire installation process (in addition to the documentation found on the Wiki), as well as they offer pre-installation support to any new releases of the software, if appropriate. This leads us to the first question: 1) Who else is interested in installing the cloud platform at their location for experimental purposes? It is planned to give technical assistance to the local installations, if needed. PSNC's documentation will be available on the Wiki, Maarten (the lead developer) will be able to give limited support and make the development image available for download, as well as new software releases will come out. Do note that at the moment Maarten is working of the tool updates according to his own agenda, therefore it would be beneficial to know what our common roadmap for the tool would be and steer the developments into that direction. So, the second question is: 2) Who is interested in introducing this cloud platform as a service and what would be the main characteristics of such a service? Obviously, the potential service models are subject to discussion. You can find some high level scenarios at https://confluence.terena.org/display/CloudStorage/Service+Delivery+Scenarios Whether NRENs are willing to offer their own storage capacity as back-end or we look for commercial cloud storage offerings, the storage API development is a key issue. Three potential options are envisaged at the moment: a) Develop a direct bridge to your storage back-end. b) Develop a generic S3 API (independent from Amazon) c) Develop a Jclouds API (to comply with many other interfaces) We have to know first who is interested in hooking up the cloud platform with their in-house storage and what those cloud storage implementations are based on. To this end, we've come to the third question: 3) Which API development option would you prefer, purely from your storage back-end and service development point of view? Please, think about these three questions and get back to me, to the list, or to the TF-Stogare meeting and discuss!!! http://www.terena.org/activities/tf-storage/ws13/
06 August 2012
Dear pilot participants, The summer period is undoubtedly a bit silent but those who are not on vacation can play with the TERENA installation of the Trusted Cloud Drive platform and collect some hands-on experience, I hope. At the moment, 14 people have access to the service and I've already received some feedback from the beta testers. I'd just like to encourage all of you to get access, play with the tool, and give us some feedback. https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN Starting from September, we can speed up the work and start looking at the potential service delivery scenarios https://confluence.terena.org/display/CloudStorage/Service+Delivery+Scenarios More importantly, the pilot participants with firm interest and dedicated efforts can join the development team and become the co-owner of the code that ensures the long-term sustainability. TF-Storage organises a meeting on 26-27 September 2012, in Dubrovnik, Croatia. http://www.terena.org/activities/tf-storage/ws13/ Part of the meeting will be dedicated to the cloud pilot discussion, initial feedback, and pilot roadmap. I'd like to invite everyone to register to the meeting and participle ether in person or remotely https://www.terena.org/events/details.php?event_id=2317
26 June 2012
Dear all, First of all, thank you very much for your interest in the TERENA Trusted Cloud Drive pilot! At the moment we have 24 people on this 'cloud-pilot' mailing list. You can check the current list of participants at https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN This list is strictly for the active pilot participants contributing to Phase ii. If you just want to be informed about the major milestones of the pilot you can join the 'storage' or the 'nren-clouds' lists of TERENA http://www.terena.org/mailinglists.php I have updated the Wiki page with two things: 1) How to get access to the TERENA installation of the service for initial testing https://confluence.terena.org/display/CloudStorage/Get+access+to+the+TERENA+installation Please, follow the instructions on the Wiki and e-mail me your id to be whitelisted by the end of this week. Do note that due to the current limitations of the setup the number of test users is limited! 2) Preliminary service delivery scenarios https://confluence.terena.org/display/CloudStorage/Service+Delivery+Scenarios One of the aims of the pilot is to design, implement, test, and validate various potential service delivery scenarios. I depicted four basic scenarios on the Wiki. The first scenario is basically the one implemented at TERENA. Other potential scenarios are subject to discussion.
18 June 2012
Dear participants, This is my very first mail to the list. I maintain this list to collect the participants (technical contacts) interested in participating the public trial phase of the TERENA Trusted Cloud Drive pilot. If you just want to be informed about the major steps and progress of the pilot, you don't have to be on this list! I'll inform the TF-Storage task force list as well as the NREN-cloud list about the milestones of the pilot so that you can follow those lists. This list is mainly for the active participants of the pilot! Please, take a look at the Wiki page of Phase ii. https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN I've added the list of interested participants there, as of today. Let me know if you need any update there! I'm still waiting for some confirmations before the list is more of less complete and I can call for the kick-off VC. Stay tuned!