SC19 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

HDF5 as a Vehicle for In Transit Data Movement


Workshop: HDF5 as a Vehicle for In Transit Data Movement

Abstract: For in transit processing, one of the fundamental challenges is the efficient movement of data from producers to consumers. We investigate the performance characteristics of an in transit data transport mechanism that leverages the HDF5 parallel I/O library and is accessible as one of several different in transit transport mechanisms in the SENSEI generic in situ framework. For in transit use cases at scale on HPC platforms, one might expect that an in transit data transport mechanism that uses faster layers of the storage hierarchy, such as DRAM memory, would always outperform a transport that uses slower layers of the storage hierarchy, such as an NVRAM-based persistent storage presented as a distributed file system. However, our results show that the performance of the transport using NVRAM is competitive with the transport that uses socket-based data movement across varying levels of producer and consumer concurrency.






Back to ISAV 2019: In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization Archive Listing


Back to Full Workshop Archive Listing