An approach to reproducibility problems related to porting software across machines and compilers.
Keeping science on keel when software moves
ABSTRACT The Slingshot interconnect designed by HPE/Cray is becoming more relevant in High-Performance Computing with its deployment on the upcoming exascale systems. In particular, it is the interconnect empowering the first exascale and highest-ranked supercomputer in the world, Frontier. It offers various features such as adaptive routing, congestion control, and isolated workloads. The deployment of newer interconnects raises questions about performance, scalability, and any potential bottlenecks as they are a critical element contributing to the scalability across nodes on these systems. In this paper, we will delve into the challenges the slingshot interconnect poses with current state-of-the-art MPI libraries. In particular, we look at the scalability performance when using slingshot across nodes. We present a comprehensive evaluation using various MPI and communication libraries including Cray MPICH, OpenMPI + UCX, RCCL, and MVAPICH2-GDR on GPUs on the Spock system, an early access cluster deployed with Slingshot and AMD MI100 GPUs, to emulate the Frontier system.
High Performance MPI over the Slingshot Interconnect: Early Experiences
Abstract: Linux Containers with the build-once run-anywhere principle have gained huge attention in the research community where portability and reproducibility are key concerns. Unlike virtual machines (VMs), containers run the underlying host OS kernel. The container filesystem can include all necessary non-default prerequisites to run the container application at unaltered performance. For this reason, containers are popular in HPC for use with parallel/MPI applications. Some use cases include also abstraction layers, e.g. MPI applications require matching of MPI version between the host and the container, and/or GPU applications require the underlying GPU drivers to be installed within the container filesystem. In short, containers can only abstract what is above the OS kernel, not below. Consequently, portability is not completely granted. This paper presents the experience of PRACE (Partnership for Advanced Computer in Europe) in supporting Singularity containers on HPC clusters and provides notes about possible approaches for deploying MPI applications using different use cases. Performance comparison between bare metal and container executions are also provided, showing a negligible overhead in the container execution.
Evaluation and Benchmarking of Singularity MPI containers on EU Research e-Infrastructure
Abstract. Apptainer (formerly known as Singularity) since its beginning implemented many of its container features with the assistance of a setuid- root program. It still supports that mode, but as of version 1.1.0 it no longer uses setuid by default. This is feasible because it now can mount squashfs filesystems, ext3 filesystems, and overlay filesystems using unprivileged user namespaces and FUSE. It also now enables unprivileged users to build containers, even without requiring system administrators to configure /etc/subuid and /etc/subgid unlike other “rootless” container systems. As a result, all the unprivileged functions can be used nested inside of another container, even if the container runtime prevents any elevated privileges. As of version 1.2.0 Apptainer also supports completely unprivileged encryption of Singularity Image Format (SIF) container files. Performance with a particularly challenging HEP benchmark using the FUSE-based mounts both with and without encryption is essentially identical to the previous methods that required elevated privileges to use the Linux kernel-based counterparts.
The modern form of containers, as popularly known through platforms like Docker, Singularity (Apptainer), Podman or Charliecloud, to cite only a few, began to take shape over a decade ago. However, the fundamentals behind containerization and actual benefits of containers in scientific computing remain largely unclear to a vast majority of users. In fact, there is a significant gap between simplistic “Hello world” examples found online and real scientific applications. Rumors suggest that achieving satisfactory performance on supercomputers across multiple nodes is impossible. This paper describes how to leverage the full potential of high-speed networks for inter-node communications with UCX on Fram and Betzy (from the Norwegian national e-infrastructure). It is then explained how to achieve near-native performance on Lumi (EuroHPC's flagship) despite its OFI "Slingshot-11" interconnect and proprietary library. Results obtained in the standard OSU Micro-benchmarks tests for latency and bandwidth and with a fully-fledged climate model show that containerized applications work just as well as "bare-metal", are portable and provide bit-for-bit reproducibility on different platforms. Containers are therefore highly recommended to minimize deployment and porting issues \emph{i)} for AaaS (Applications as a Service) coming with all the necessary software environment (rather than source code only); and \emph{ii)} so that HPC users do not have to rely on anybody to install what they need and can be operational within minutes whilst still getting top performance.
Unveiling the Potential of Containers in Scientific Computing
Fouilloux, Anne, and Jean Iaquinta. "Unveiling the Potential of Containers in Scientific Computing." ROHub. Feb 24 ,2024. https://w3id.org/ro-id/18d3e6d6-e559-4e43-b464-55275bd14708.
SLURM job scripts to submit jobs on HPCs.
Container definition files for singularity/Apptainer.
Folder containing links to scientific papers and other articles used in the study.
Source code for various tools used in this work
Singularity image files to ensure reproducibility.
Slurm script to submit the OSU latency and get bandwidth tests with 2 CPUs on different nodes on Lumi using an OpenSUSE/Leap:15.3 container with MPIch4.1.2 and the OFI libraries from the host
This is a definition file to illustrate the use of an OpenSUSE/Leap:15.3 container with MPIch4.1.2 and support for UCX1.15.0 to run the OSU Micro-benchmarks tests version 7.3
OpenSUSE/Leap:15.3 container with MPIch4.1.2 and support for UCX1.15.0 to run the OSU Micro-benchmarks tests version 7.3
Slurm script to submit the OSU latency and get bandwidth tests with 2 CPUs on one node on Fram with the simple container
Container with all the necessary packages to run CESM2.3 with the OFI-swap method
OpenSUSE/Leap:15.3 container with MPIch4.1.2 and a "dummy" LibFabric1.14.0 to run the OSU Micro-benchmarks tests version 7.3
Singularity Image file for a simple Ubuntu:22.04 container with MPIch4.1.2 to run the OSU Micro-benchmarks tests version 7.3
Slurm script to submit the a job on Lumi using the OpenSUSE/Leap:15.3 container with MPIch4.1.2 (like to prepare or run an ESM) and swapping the OFI for that on the host at runtime
This is a definition file to illustrate the use of an OpenSUSE/Leap:15.3 container with MPIch4.1.2 and a "dummy" LibFabric1.14.0 to run CESM2.3 version alpha 17
This is a definition file to illustrate the use of a simple Ubuntu:22.04 container with MPIch4.1.2 to run the OSU Micro-benchmarks tests version 7.3
This is a definition file to illustrate the use of an OpenSUSE/Leap:15.3 container with MPIch4.1.2 and a "dummy" LibFabric1.14.0 to run the OSU Micro-benchmarks tests version 7.3
Community Earth System Model version 2.3 Alpha 17
Slurm script to submit the OSU latency and get bandwidth tests with 2 CPUs on different nodes on Fram with the simple container
NICEST-2 - the second phase of the Nordic Collaboration on e-Infrastructures for Earth System Modeling focuses on strengthening the Nordic position within climate modeling by leveraging, reinforcing and complementing ongoing initiatives.
Nordic Collaboration on e-Infrastructures for Earth System Modeling Tools
The modern form of containers, as popularly known through platforms like Docker, Singularity (Apptainer), Podman or Charliecloud, to cite only a few, began to take shape over a decade ago.
Science and technology/Technology and engineering/IT-computer sciences
Unveiling the Potential of Containers in Scientific Computing.
Economy, business and finance/Economic sector/Computing and information technology/Hardware
Containers are therefore highly recommended to minimize deployment and porting issues \emph{i)} for AaaS (Applications as a Service) coming with all the necessary software environment (rather than source code only); and \emph{ii)} so that HPC users do not have to rely on anybody to install what they need and can be operational within minutes whilst still getting top performance.
Arts, culture and entertainment/Mass media/Television