HDF5 as a Vehicle for In Transit Data Movement
TimeMonday, 18 November 201911:42am - 12pm
DescriptionFor in transit processing, one of the fundamental challenges is the efficient movement of data from producers to consumers. We investigate the performance characteristics of an in transit data transport mechanism that leverages the HDF5 parallel I/O library and is accessible as one of several different in transit transport mechanisms in the SENSEI generic in situ framework. For in transit use cases at scale on HPC platforms, one might expect that an in transit data transport mechanism that uses faster layers of the storage hierarchy, such as DRAM memory, would always outperform a transport that uses slower layers of the storage hierarchy, such as an NVRAM-based persistent storage presented as a distributed file system. However, our results show that the performance of the transport using NVRAM is competitive with the transport that uses socket-based data movement across varying levels of producer and consumer concurrency.