A bridge between GridFTP and HPSS

12.02.14 -

By Nicole Gaynor

Big data requires new tools for transferring data between machines, especially when the source and target machines use different data transfer protocols. The GridFTP High Performance Storage System data storage interface (GridFTP HPSS DSI), developed at the National Center for Supercomputing Applications, is exactly that.

HPSS, a highly scalable, hierarchical storage system, and GridFTP, a newer transfer protocol, use different types of commands. The HPSS DSI translates communications between the GridFTP protocol and the HPSS API (application programming interface, the set of commands HPSS understands).

“With this enabling piece of software, existing and new HPSS installations can join with grid infrastructure as first-class citizens and leverage emerging transfer solutions such as Globus Online,” says Jason Alt, developer of the HPSS DSI and programmer at NCSA.

The new Blue Waters Data Sharing Service is only one of myriad possible implementations of the HPSS DSI. The service will allow current and former supercomputer users to share their research data with colleagues who do not have access to the supercomputer.

More information on the HPSS DSI and the code base is available at https://github.com/JasonAlt/GridFTP-DSI-for-HPSS.

National Science Foundation

Blue Waters is supported by the National Science Foundation through awards ACI-0725070 and ACI-1238993.