11 March 2013
Dear TCD pilot participants, all, This is my regular e-mail update on the progress of the TERENA Trusted Cloud Drive pilot. You can always take a look at the latest updates at https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN My last update was in December 2012, my apologies for the long radio silence. The actual figures of the pilot are as follows: - The total number of platform test/dev installations has raised to 9 (+2): TERENA, PSNC, BELNET, CESNET, GRNET, FCCN, Srce/CARNet, RNP, CESCA - 46 user accounts (+0) are enabled on the TERENA pilot service at https://tc2.terena.org - The cloud-pilot mailing list has 62 members (+1). The small increase in these figures during the last 3 months means that the interest generated by the pilot within the TERENA community has reached its maximum and stabilised around those numbers. *Major milestones* As it was reported at the last TF-Storage meeting in Berlin (take a look at my slides http://www.terena.org/activities/tf-storage/ws14/slides/2013036-TCDpilot.pdf) the pilot has reached several major milestones: 1) CESNET (Jakub Peisar) has done the performance/scalability tests of the TCD platform, the results and conclusions can be found at https://confluence.terena.org/display/CloudStorage/Software+tests 2) GRNET (Christos Loverdos) has implemented the TCD integration with Pithos+ back-end. The code has been contributed to the TERENA Github and the work was successfully demonstrated to TF-Storage in Berlin http://www.terena.org/activities/tf-storage/ws14/slides/20130306-Pithos-CloudDrive.pdf 3) RNP (Gui Maluf) has implemented the OpenStack Object Storage (Swift) S3 API to the TCD platform. The code has been contributed to the TERENA Github https://github.com/TERENA/CloudDrive It is planned to integrate TCD (via the OpenStack S3 API) to RNP's cloud storage infrastructure. 4) CESCA (Lorenzo J. Cubero) is progressing with the Jclouds API implementation to the TCD platform. Progress will soon be reported. These software developments and platform integration efforts allow us to focus on the service development and validation aspects of the pilot in its last month (until April 2013)! On behalf of the community, TERENA very much appreciates the work committed by these people and their organisations. *TCD platform directions* Note that Maarten Koopmans (Vrijheid.net), the lead developer of the platform, informed TERENA that he's going to consolidate the code base; better Scala, less dependencies, smaller code size are expected. The main Github branch will be closed however the TERENA Gitghub fork remains open. Concerning the features and functions of the TCD platform the following strategic direction has been envisaged by TERENA: - TCD should focus on its main asset; maintaining trust and privacy by separating metadata and encryption keys (kept in a trusted place) from the storage data (exported to back-end storage). - TCD should not compete with the feature-rich front-end applications available on the market but broker them to various storage back-ends. - Therefore, TCD should be a thin layer (controlled and operated by NRENs) separating/interfacing end-user application domains and cloud service provider domains; considered to be a "storage middleware". - The integration of TCD should be considered with both end-user oriented storage applications and back-end storage cloud provider platforms. *Use cases and service validation scenarios* In line with the strategic direction outlined above, the pilot will further explore: A) the service brokering scenarios made available by the TCD-Pithos+ and the TCD-Swift integrations; different NREN storage infrastructures and/or public cloud storage back-ends can be brokered down to user domains in a trusted, privacy-controlled way. B) the service integration scenarios with feature-rich end-user storage applications; both community-developed and commercial solutions (such as OwnCloud and PowerFolder) will be investigated for potential integration with TCD functionality to preserve trust and privacy. That's it for now! I'd like to encourage everyone to contribute to the service development and validation aspects of the pilot in its very last month.
...
17 December 2012
Dear TCD pilot participants, all, This is my regular e-mail update on the progress of the TERENA Trusted Cloud Drive pilot (the last one in 2012) however, you can always take a look at the latest updates at https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN The last pilot conf.call was on 14 November 2012, the minutes can be found at https://confluence.terena.org/display/CloudStorage/Meeting+notes Since then, FCCN (Portugal), CESNET (Czech Rep.) and RNP (Brazil) managed to install a service platform instance for experimental purposes at their locations. The total number of platform test installations has raised to 7. 43 user accounts are enabled on the TERENA pilot service https://tc2.terena.org. The cloud-pilot mailing list has 62 members. Thanks to CESNET, the Installation Guide - maintained by the community - has been significantly improved with useful fixes, tips and tricks. Please, take a look at https://confluence.terena.org/display/CloudStorage/Installation+Guide The various software tests (performance, security, etc) are on-going at CESNET and PSNC. Feel free to offer any software testing effort to the pilot. The voluntary software developers have been working on the platform compatibility improvements, namely: - Lorenzo (CESCA) on the Jclouds back-end API - Gui (RNP) on the OpenStack Swift integration - Christos (GRNET) on the Pithos+ interworking I would like to *encourage everyone* to post questions, updates, useful hints on the mailing list of the pilot. TERENA has created an Github account (as virtual organisation) https://github.com/terena and forked CloudDrive as a project. Developers can be added as members of the organisation administered by TERENA. The main CloudDrive branch is still maintained by Maarten and we try to keep up the consistency as the development efforts are converging. There are plans to initiate a dialogue (and potential partnership) with the Helix Nebula http://helix-nebula.eu/ project as well as the forthcoming GN3+ project could related (SA7) activity. This is still in embryonic phase, we have to see the potentials and the level of uptake of TCD in the community. TERENA has recently received some interest from commercials willing to provide various cloud services to the R&E community. TERENA TF-Storage group had a discussion with Joyent https://www.terena.org/events/details.php?event_id=2473 where TCD was mentioned as a potential service platform brokering Joyent's IaaS offerings to the community. TERENA will continue the discussion with Joyent in early 2013. Keep up the good work in 2013! Season's greetings, Peter
...
14 November 2012
Participants:
Panos Louridas, GRNET
Christos KK Loverdos, GRNET
Lorenzo J. Cubero, CESCA
Gui Maluf, RNP
Peter Szegedi, TERENA
NOTES
Previously (via e-mail)1) Lorenzo (CESCA) has offered to work on the Jclouds API implementation and testing at the platform back-end. He has been talking to Adrian Cole [jcloud founder] recently. http://www.jclouds.org/documentation/reference/supported-providers/
2) Gui (RNP) has offered to make a bridge between OpenStack Swift and TERENA cloud broker. AFAIK, RNP has an OpenStack based cloud storage infrastructure. http://docs.openstack.org/trunk/openstack-object-storage/admin/content/configuring-openstack-object-storage-with-s3_api.html
3) Christos (GRNET) has just offered to make a bridge between Pithos+ (OpenStack based) and the TERENA cloud broker. He has been working with GRNET on Okeanos - their overall cloud storage service - and more specifically on its accounting components, written in Scala. He is an expert in Scala.
Lorenzo has started to work on the Jclouds API implementation. He also managed
to install the platform at his computer using the VMware images available. He is
now trying to install it right from Github, manually. Gui has failed with the first
attempt, and he is seeking for help on the mailing list. Christos will try and install
the platform at GRNET, too.
The very first Milestone to Lorenzo, Gui and Christos would be to get the development platform up and running at their location.
Panos asked about the code license and the Github account. Peter said that the license is
Apache Licenses, Version 2.0. The Github account is owned by the lead developer (Maarten Koopmans)
at the moment.
Action on Peter (TERENA) to talk to Maarten about the migration of the Github account to TERENA.
Participants agreed to coordinate efforts and work together towards the common goal. TERENA will
facilitate this development work by providing the mailing list, common wiki space, and access to
the Github account.
Next development meeting will be called as soon as the first Milestone has been reached.14 November 2013
...
...
26 October 2012
PARTICIPANTS: Jean-Philippe Evrard, Belnet Jakub Peisar, CESNET Maarten Koopmans Szabolcs Szekelyi, NIIF Maciej Brzezniak, PSNC Lorenzo J. Cubero, CESNETCESCA Peter Szegedi, TERENA Joël Casutt, SWITCH
Roberto Araujo, RNP
Panos Louridas, GRNET
If I missed you and you want to be minuted, let me know! NOTES: 1) *Pilot status update* Peter (TERENA): The source code, its documentation, the Installation Guide made by PSNC, and the downloadable VMware images (made by Maarten) are all available on the Wiki https://confluence.terena.org/display/CloudStorage/Phase+II+-+OPEN The pilot is running until March 2013 (five months left). The aims of the pilot are: a) to ensure the sustainability of the service platform (i.e. gather the developers' community around the code that can implement the roadmap (new features, bug fixes) agreed). b) to ensure the required scalability, performance, and stability of the code (i.e. run coordinated tests on several implementations of the platform and come up with recommendations for improvements). c) to investigate the potential service delivery models using the cloud storage broker platform (i.e. collect use cases, make a list of necessary features to be implemented, and agree on a common roadmap). As of today, TERENA and PSNC have already installed 1-1 service instances, and BELNET, CESNET, Scre, FCCN and CESCA signaled that they also plan to install the platform at their location. This is considered as the core group. There are >25 test accounts provisioned to the TERENA installation so far. This group is considered as the core of alpha/beta testers form a user perspective. 2) *Software tests* Maarten: Previously, the platform has been tested on EC2 resources. Three virtual servers have been used as a redundant Metadata store and two additional virtual servers for WebDAV and Website. This is the recommended setup for testing. Maciej (PSNC): PSNC will start with the security tests first and do a code review. Later on performance and scalability tests (e.g., stresstest) will also be done. In case of interest, PSNC plan to test various user applications on top of WebDAV (maybe next year). Jean-Philippe (Belnet): The platform has been installed on one single server at BELNET. Once the on-going storage infrastructure tendering is over, Belnet plans to do some test too. The software tests must be coordinated. Jakub (CESNET): CESNET will also install the platform on several virtual servers as suggested. Performance test (starting with the Metadata store) will then be done in a well-documented way. *ACTION* 1 on Peter (TERENA) to coordinate and compile a list of test procedures and call for potential testers on the mailing list. Maarten commented that the file level compression and encryption is configurable (can be turned off) that can immediately improve the performance with a factor, if needed. The Voldemort metadata store is also linearly scaling, additional virtual servers can be added as the metadata increases. 3) *Software development* Maarten: Replication/Synchronisation between two or more data storage back-ends is not implemented at the moment and it is not the primary aim. Szabolcs (NIIF): It is easier to implement replication at the back-end storage level rather than in the broker platform on top. It is recommended NOT to deal with the replication at the platform level. Maarten: The implementation of "sharing" was in the original development roadmap, but Maarten is fully booked by the end of the year with other projects so, no progress is expected from their side. Peter: An other necessary feature would be an administrative web interface. That can help all the service adopters to configure the platform easily. This development can be done by a web developer, if agreed. Peter (TERENA): In principle, the service development procedure would be that the pilot participants come up with an agreed list of features to be implemented, and then the actual coders (Maarten or others) will be selected once the roadmap is clear. Lorenzo (CESCA): Lorenzo volunteered to contribute to the pilot and implement a Jclouds back-end API as well as do some performance tests on it. The idea behind Jclouds API is to ensure the full flexibility of the platform in terms of storage back-ends. In theory, Jclouds makes it possible to configure the actual storage API used. *ACTION* 2 on Lorenzo (CESCA) to start working on the Jclouds implementation ASAP and report back to the pilot participants by the end of the year. Lorenzo also asked if a flexible front-end API (e.g., REST API) would be of interest. Peter answered that the priority would be to ensure the largest back-end API compatibility possible. NRENs are looking for solutions to be put on top of their storage infrastructure (using different APIs) and provide services to their users quick and easy. The meeting was closed. No agreement on the next call, to be clarified on the mailing list. LIST OF ACTIONS: ACTION 1 on Peter (TERENA) to coordinate and compile a list of test procedures and call for potential testers on the mailing list. ACTION 2 on Lorenzo (CESCA) to start working on the Jclouds implementation ASAP and report back to the pilot participants by the end of the year.
...