The HECToR Service is now closed and has been superceded by ARCHER.

Direct Numerical Simulation (DNS) of turbulent fluid flows - II

This Distributed Computational Science and Engineering (dCSE) project is to further develop Incompact3D, an incompressible Navier-Stokes solver, and its sister codes, Compact3D and QuasiCompact3d, for compressible flow simulations. These applications are used by the Turbulence, Mixing and Flow Control group at Imperial College and its academic collaborators to conduct state-of-the-art turbulence studies. In the very successful first Incompact3D project, the scalability of the code was significantly improved via the creation of the 2DECOMP&FFT library, which is a 2D pencil decomposition communication framework with a distributed FFT interface. As a result, Incompact3D now regularly uses 4,000-16,000 cores on HECToR for production runs.

There are two aims of this project: firstly, to enable the efficient parallel simulation of large-scale compressible turbulent flows; and secondly to enable the embedding of explicit numerical schemes within the framework of a spatially implicit code for studying a wider range of problems, such as particle-turbulence interaction. This will be achieved by the following work:

  • To parallelise the sister codes of Incompact3D (Compact3D and QuasiCompact3d). These codes will require a full parallelisation starting from their serial implementations. 2DECOMP&FFT framework will be used for the parallelisation.
  • To develop a new communication module for the 2DECOMP&FFT library to enable the efficient embedding of explicit numerical schemes within the framework of a spatially implicit code.

The individual achievements of the project are summarised below:

  • General improvements of the codes to use modern Fortran features.
  • Enabling the reuse of key subroutines across the family of codes to improve maintainability.
  • Parallelasion of the serial Compact3D and QuasiCompact3d using 2DECOMP&FFT. A modern code development procedure was established to help future development.
  • Parallel code performance was demonstrated. For a typical 3D mesh of size 151×203×151 a parallel efficiency of nearly 80% can be achieved between 32 and 128 cores on HECToR phase 3.
  • A general communication routine was implemented in 2DECOMP&FFT to enable halo-cell communication between neighbouring blocks within the framework of transpose-based parallel codes. Any number of halo-cell overlaps are supported.
  • The new communication code is being used to implement key algorithms within Incompact3D, such as the 3D interpolations required by the Immersed Boundary method to prescribe boundary conditions on the fluid-solid interface, and similar interpolation schemes required to implement particle tracking.

Please see PDF or HTML for a report which summarises this project.