Lustre Community BoF: Lustre in HPC, AI, and the Cloud

Authors: Stephen Simms (Indiana University), Frank Baetke (European Open File System Association (EOFS))

Abstract: Lustre is the leading open-source and open-development file system for HPC. Around two thirds of the top 100 supercomputers use Lustre. It is a community developed technology with contributors from around the world.

Lustre currently supports many HPC infrastructures beyond scientific research, such as financial services, energy, manufacturing and life sciences. Lustre clients are available for a variety of technologies, such as x86, POWER, and Arm. At this BoF, Lustre developers, administrators, and solution providers will gather to discuss recent Lustre developments and challenges, including the role of Lustre in AI and its use in Cloud environments.

Long Description: Lustre is the leading open-source and open-development file system for HPC. Around two thirds of the top 100 supercomputers use Lustre file systems. Lustre today is a community developed file system with contributors from around the world. Lustre supports many HPC infrastructures beyond its traditional stronghold of scientific research including financial services, energy, manufacturing, life sciences and animation and Lustre clients are available for x86, POWER, and Arm.

At this BOF, Lustre developers, administrators, and solution providers will gather to discuss recent developments, such as Persistent Client Caching (PCC) and Data on Metadata (DoM) and new challenges and corresponding opportunities, including the role of Lustre in AI and its use in Cloud environments.

Today Lustre is one of the most widely adopted technologies in HPC, from the University of Cambridge’s Data Accelerator system to the Tianhe-2 system at the National Supercomputer Center in China, and the Frontera system at TACC, to the K computer at the RIKEN Advanced Institute for Computer Science in Kobe, Japan, and more. Lustre is also widely used across mid- and small-scale HPC systems with continued adoption attributable to the stability of the Lustre file system as it has matured.

Vital to the technology is the community that continues to drive Lustre forward. As Lustre has evolved into a true open-source and open-development model, the end users, developers, and solutions providers have come together through the worldwide OpenSFS and EOFS communities.

This community development model has resulted in significant new features, improved stability and broader adoption.

At the 2019 Lustre BOF we will focus on feature developments and discuss how they will shape future Lustre deployments. Across all key-application segments, Lustre is at the heart of many HPC infrastructures and must continue to evolve in order to support new use cases, such as AI and Cloud environments. We will explore these use cases and discuss the Lustre roadmap for meeting the requirements they present.

The session will include a town hall meeting open to participants. OpenSFS and EOFS will provide initial discussion topics focusing on a brief set of challenges. The list of challenges will be compiled by the OpenSFS and EOFS by polling the community before the BOF.

The Lustre BOF has been held at previous SC events and has been well attended and enthusiastically received. SC presents a rare opportunity for the worldwide Lustre community to meet and discuss how to make Lustre more successful. Attendance is usually very good with well over 150 attendees. The Lustre BOF at SC17 had about 190 attendees and the BOF at SC18 had approximately 180.

The targeted audience includes all who are involved with Lustre deployments such as administrators, system architects, developers, solution providers and end users. It includes all who have been contributing or reviewing new features or providing additional tools for management and control.

A written report summarizing the Lustre Community BOF will be developed and will be made available to the Lustre community.

URL: http://opensfs.org


Back to Birds of a Feather Archive Listing