From SC Papers to Student Cluster Competition Benchmarks: Joining Forces to Promote Reproducibility in HPC
The tradition of collaboration between the Technical Program and Students@SC is stronger than ever at SC19. This year we mark the fourth Reproducibility Challenge in the Student Cluster Competition (SCC) that brings together these two conference components to select a paper from SC18 and adapt its artifacts to become one of the SC19 SCC benchmarks.
The Selection of an SC18 Paper
We are excited to announce that the paper selected to serve as the SCC benchmark for the Reproducibility Challenge this year is “Computing Planetary Interior Normal Modes with a Highly Parallel Polynomial Filtering Eigensolver” by Jia Shi, Ruipeng Li, Yuanzhe Xi, Yousef Saad, and Maarten V. de Hoop. A team of reviewers selected this paper from 40 finalists based on the paper’s Artifact Descriptor (AD) and its suitability to the SCC. The authors have been working with us to create a reproducible benchmark that builds on the paper’s results. At SC19, the 16 SCC teams will be asked to run the benchmark, replicating the findings from the original paper under different settings and with different datasets.
What makes the work of the 16 student teams particularly relevant is the replication of the paper’s work across 16 different clusters. In the era of heterogeneous computing, porting applications from one platform to another is not a simple task. The work of the 16 teams at SC19 is a fantastic way to dive into reproducibility challenges across heterogeneous platforms and emerge with shareable, robust insights. It is the ensemble of 16 solutions on 16 different platforms executed by 16 different teams that will earn this paper ACM’s “Results Replicated” badge in the ACM digital library. Sharing is at the core of the Reproducibility Challenge – so, the work of the SCC teams will be collected into a special issue of the journal Parallel Computing. We have already published two special issues (SC16, SC17) about this effort in recent years. These reports are a must for system administrators and performance experts.
Behind the Scenes of Putting Together the SC19 SCC Benchmark
Many volunteers participate in the selection of the paper, the creation of the benchmark for the SCC teams, the assessment of the students’ work, and the publication of the special journal issue.
A first round of reviews was done to determine feasibility for the competition. Here, we looked at whether the finalist papers had an application that could be run by the student teams on the broad range of hardware types and cluster configurations that are typically fielded by SCC teams. This initial review eliminated nearly 80% of the potential papers because, for example, they used proprietary compilers or ran only on specific hardware.
A second round of reviews looked for which application would be best suited for the SCC teams. For this round, each paper had at least three reviews. Following the review scoring, the committee ranked the submissions based on criteria such as the application’s concrete, real-world impact, and the student experience while working with the benchmark. Discussions with the authors of the finalist papers focused on the feasibility of adapting their applications to the Student Cluster Competition in cases where the information was not readily available in the submitted paper. Following these interviews, the committee met to determine which application to invite.
The selection of the paper is just one part of a long process that will end with the preparation of one of the SCC benchmarks – the one for the Reproducibility Challenge. The benchmark will be revealed at SC19. Our work will continue after SC19 with the publication of a new special issue of Parallel Computing featuring the students’ reports from the SC19 SCC Reproducibility Challenge.
Mark Your Calendar
The Student Cluster Competition will be held Monday–Wednesday, November 18–20, 2019. Visit the SCC booth in the Exhibits Hall at SC19 and chat with students about the Reproducibility Challenge. Celebrate the student participants and the authors of the selected paper at the Awards Ceremony on Thursday of the conference. And don’t miss the next special issue of Parallel Computing containing the SCC reports. Join us in Denver to discover why HPC is now!
——
Scott Michael, Reproducibility Liaison for Students@SC
Stephen Harrell, SC19 Students@SC Chair
