File Transfer

A common problem for many scientific applications is the replication of - often large - data sets (files) from one system to another. (For the generalized problem of transferring data sets from a source to multiple destinations, see DataDissemination.) Typically this requires reliable transfer (protection against transmission errors) such as provided by TCP, typically access control based on some sort of authentication, and sometimes confidentiality against eavesdroppers, which can be provided by encryption. There are many protocols that can be used for file transfer, some of which are outlined here.

Several high-performance file transfer protocols are used in the Grid community. The "comparative evaluation..." paper in the references compares FOBS, RBUDP, UDT, and bbFTP. Other protocols include GridFTP, Tsunami and FDT. The eVLBI community uses file transfer tools from the Mark5 software suite: File2Net and Net2File. The ESnet "Fasterdata" knowledge base has a very nice section on Data Transfer Tools, providing both general background information and information about several specific tools. Another useful document is Harry Mangalam's How to transfer large amounts of data via network�a nicely written general introduction to the problem of moving data with many usage examples of specialized tools, including performance numbers and tuning hints.

Network File Systems

Another possibility of exchanging files over the network involves networked file systems, which make remote files transparently accessible in a local system's normal file namespace. Examples for such file systems are:

References

– Main.TobyRodwell - 2005-02-28

– Main.SimonLeinen - 2005-06-26 - 2015-04-22