The HECToR Service is now closed and has been superceded by ARCHER.

Direct Numerical Simulations (DNS) of Turbulent Fluid Flows

This project was to modernise Incompact3D, a CFD application for Direct and Large-eddy simulations. Incompact3D is used by the Turbulence, Mixing and Flow Control group at Imperial College London to conduct cutting-edge turbulence studies.

One recent application of Incompact3D is to complement experimental results from wind tunnel measurements for new flow concepts concerning turbulence generated by multi-scale/fractal objects. This class of new flow concepts is offering possibilities for brand-new solutions useful in industrial mixers, silent air-brakes, new ventilation and combustion devices. To resolve the turbulent eddies associated with the smallest scale wind tunnel measurements, very high resolution multiscale flow simulations are required in order to understand the underlying physics.

Incompact3D solves the governing equations of incompressible flows. The application uses a 6th order compact finite difference scheme to discretize the convective and diffusive terms of the Navier-Stokes equations on a Cartesian mesh. It also solves a pressure Poisson's equation to enforce mass conservation. Both spatially implicit numerical schemes require major code development to properly exploit the facilities of a supercomputer. This report will highlight key areas that are relevant to an efficient code parallelisation.

The overall outcome of this work may be summarised as follows:

  • An upgrade of Incompact3D's domain decomposition algorithm from a 1D slab decomposition to a 2D pencil decomposition, with the following benefits:
    • Lift scalability restriction - with the old 1D decomposition a typical simulation using a 2048*515*512 mesh could only use up to 512 cores. With the new 2D decomposition, meshes with up to 4096*4096*4096 grid points can now utilise up to 16384 cores on HECToR and the upper limit of the 2D decomposition strategy is now at hundreds of thousands of cores.
    • Productivity improvement - a typical simulation using a 2048*515*512 mesh with 512 cores on HECToR phase 1 required a runtime of 25 days (wall-clock time); now, this size of mesh can utilise 8192 cores efficiently leading to a run time of less than 2 days.
    • Efficiency savings - for representative sizes the new Incompact3D is about 20% faster than the old code. This will result in a saving of 800,000 HECToR AUs (assuming 4 million AUs from the PI's EPSRC grant and UKTC's HECToR allocation are to be consumed).
  • The creation of the 2DECOMP&FFT library, encapsulating all the reusable software components.
    • A general-purpose 2D decomposition library and a portable and highly scalable programming interface enabling distributed 3D FFT calculations.
    • The library is freely available to all HECToR users and will benefit scientific applications that require a 2D decomposition or performance upgrades for their distributed FFT calculations.
    • Several scientific applications are being (or to be) updated using 2DECOMP&FFT including codes in combustion, ocean modelling and compressible CFD.
  • A fully spectral FFT-based pressure Poisson solver.
  • Shared-memory optimisations for multi-core nodes via System V IPC API.
  • Parallel I/O implementation using MPI-IO.
  • A paper at Cray User Group 2010 conference.

Please see PDF or HTML for a report which summarises this project.