SC19 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

Poster 78: Understanding HPC Application I/O Behavior Using System Level Statistics


Authors: Arnab K. Paul (Virginia Tech), Olaf Faaland (Lawrence Livermore National Laboratory), Adam Moody (Lawrence Livermore National Laboratory), Elsa Gonsiorowski (Lawrence Livermore National Laboratory), Kathryn Mohror (Lawrence Livermore National Laboratory), Ali R. Butt (Virginia Tech)

Abstract: The processor performance of high performance computing (HPC) systems is increasing at a much higher rate than storage performance. Storage and file system designers therefore require a deep understanding of how HPC application I/O behavior affects current storage system installations in order to improve storage performance. In this work, we contribute to this understanding using application-agnostic file system statistics gathered on compute nodes as well as metadata and object storage file system servers. We analyze file system statistics of more than 4 million jobs over a period of three years on two systems at Lawrence Livermore National Laboratory that include a 15 PiB Lustre file system for storage. Some key observations in our study show that more than 65% HPC users perform significant I/O which are mostly writes; and less than 22% of HPC users who submit write-intensive jobs perform efficient writes to the file system.

Best Poster Finalist (BP): no

Poster: PDF
Poster summary: PDF


Back to Poster Archive Listing