bbFTP
bbFTP can be used to efficiently transfer large amounts of data to/from HECToR. This document provides a brief introduction to using bbFTP on HECToR. More detailed information and documentation is available on the bbFTP website.
bbFTP is one of the tools we recommend if you have large amounts of data to transfer to or from HECToR.
bbFTP is an open-source secure multi-streamed file transfer utility, released under the GNU General Public License. It was written by Gilles Farrache at IN2P3 Computing Center in Lyon, France. bbFTP has been optimised for the transfer of large files: it securely authenticates, but then transfers file data in multiple unencrypted streams, thereby avoiding the computational overhead of encryption and decryption. bbFTP also allows large windows as defined in RFC1323, and therefore is well suited to the problem. Previous studies of grid-ftp have shown that there is good scope for improvement over scp performance. grid-ftp can be difficult to install and by default, requires a non-trivial certificate based authentication mechanism that is often too large a barrier for user uptake. bbFTP is a simpler alternative that uses password or public key based authentication in the same manner as scp, so is immediately accessible to users with minimal setup costs. However if you already have access to grid-ftp this is an equally good alternative.
Limitations
bbFTP can only transfer files, not directory trees. If you want to transfer a recursive directory tree then you first need to archive your directory structure into a single file (using, for example tar) and then transfer this archive. Once transferred, the archive can be extracted.
How to use bbFTP
There are a number of ways to initiating a bbftp transfer so for simplicity we concentrate on transferring data between HECToR and YOURMACHINE.
To "get" data from YOURMACHINE onto HECToR, issuing the command from HECToR itself:
/usr/local/packages/bbftp/bbftp -s -E remote-bbftpd-path -m -p 10 -e "get remote_filename.dat local_filename.dat" login@YOURMACHINE.ac.uk
The remote filename is relative to the user's home directory on your machine unless it begins with a "/" in which case it is interpreted as an absolute filename.
The local filename is relative to the directory in which the bbFTP command was issued unless it begins with "/" in which case it is interpreted as an absolute filename.
When run using this single command, the bbFTP executable is launched at the local and remote sites simultaneously, and the transfer is initiated over 10 parallel streams (as indicated by the -m and -p flags). We do not recommend or anticipate the need for the use of more than 10 parallel streams.
Note that the -E flag specifies the location to the bbFTP executable on the remote site.
To "get" all files in a subdirectory on YOURMACHINE onto HECToR (issuing the command from HECToR):
/usr/local/packages/bbftp/bbftp -s -E remote-bbftpd-path -m -p 10 -e "mget remote_dir/* local_dir" login@YOURMACHINE.ac.uk
This will transfer all the files in "remote_dir" into the local directory "local_dir". The same rules on relative and absolute filenames apply as specified in the previous example.
To "put" a file from HECToR onto YOURMACHINE (issuing the command from HECToR):
/usr/local/packages/bbftp/bbftp -s -E remote-bbftpd-path -m -V -p 10 -e "put local_file.dat remote_file.dat" login@YOURMACHINE.ac.uk
Once again, all the rules for relative and absolute filenames apply as in the examples above. There is also the command "mput" for transferring more than one file.
If you are using a remote bbftp client to connector to hector you should connect to login11b.hector.ac.uk and specify /usr/local/packages/bbftp/bbftpd as the server location.
Note - you will be unable to use bbftp to transfer data to/from the UK RDF filesystem via node login11b.hector.ac.uk. The RDF filesystems are not mounted on login11b.hector.ac.uk.
If you wish to transfer information to and from your home site, you should ensure the bbFTP software is also installed there. bbFTP, along with a full explanation of its usage and modes of operation can be found at the bbFTP website.
