Presentation
Balsam: Near Real-Time Experimental Data Analysis on Supercomputers
Event Type
Workshop
W
Experimental Datasets
High-fidelity Simulations
Scalable Computing
TimeMonday, 18 November 20194:45pm - 5:10pm
Location506
DescriptionThe growing sophistication of experimental data analysis at DOE scientific user facilities is poised to outstrip local resources for storage and computing. At the same time, experimental facilities generally underuse DOE supercomputers for analysis because 1) logging into a supercomputer to schedule computations complicates the experimental workflow, and 2) scientists must often interpret data on-the-fly and cannot abide by batch queueing policies. In this work, we describe how the Balsam edge service has enabled near-real-time analysis of data collected at the Advanced Photon Source (APS) with an X-ray Photon Correlation Spectroscopy (XPCS) application running on Argonne Leadership Computing Facility (ALCF) resources. Balsam provides a uniform API for remote job submission and can leverage transfer protocols like Globus to orchestrate data movement. The Balsam service then interfaces with the system scheduler to perform automated job submission and process tasks in a high-throughput, fault-tolerant fashion. We highlight the effectiveness of backfill queues when used with Balsam to process XPCS data on idle nodes with relatively low latency.
Archive
