Distributed-memory simulations of turbulent flows on modern GPU systems using an adaptive pencil decomposition library

Joshua Romero, Pedro Costa, Massimiliano Fatica

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

Abstract

This paper presents a performance analysis of pencil domain decomposition methodologies for three-dimensional Computational Fluid Dynamics (CFD) codes for turbulence simulations, on several large GPU-accelerated clusters. The performance was assessed for the numerical solution of the Navier-Stokes equations in two codes which require the calculation of Fast-Fourier Transforms (FFT): a tri-periodic pseudo-spectral solver for isotropic turbulence, and a finite-difference solver for canonical turbulent flows, where the FFTs are used in its Poisson solver. Both codes use a newly developed transpose library that automatically determines the optimal domain decomposition and communication backend on each system. We compared the performance across systems with very different node topologies and available network bandwidth, to show how these characteristics impact decomposition selection for best performance. Additionally, we assessed the performance of several communication libraries available on these systems, such as Open-MPI, IBM Spectrum MPI, Cray MPI, the NVIDIA Collective Communication Library (NCCL), and NVSHMEM. Our results show that the optimal combination of communication backend and domain decomposition is highly system-dependent, and that the adaptive decomposition library is key in ensuring efficient resource usage with minimal user effort.

Original languageEnglish
Title of host publicationProceedings of the Platform for Advanced Scientific Computing Conference, PASC 2022
PublisherAssociation for Computing Machinery (ACM)
ISBN (Electronic)9781450394109
DOIs
Publication statusPublished - 2022
Externally publishedYes
Event2022 Platform for Advanced Scientific Computing Conference, PASC 2022 - Basel, Switzerland
Duration: 27 Jun 202229 Jun 2022

Conference

Conference2022 Platform for Advanced Scientific Computing Conference, PASC 2022
Country/TerritorySwitzerland
CityBasel
Period27/06/2229/06/22

Keywords

  • computational fluid dynamics
  • direct numerical simulation
  • GPU accelerated systems
  • parallel transpose

Fingerprint

Dive into the research topics of 'Distributed-memory simulations of turbulent flows on modern GPU systems using an adaptive pencil decomposition library'. Together they form a unique fingerprint.

Cite this