The HECToR Service is now closed and has been superceded by ARCHER.

HECToR enabled Step Change in Turbulent Multiphase Combustion Simulations

The DSTAR code is used for Direct Numerical Simulation (DNS) of multiphase reactive flows in which complex interactions exist among vortex dynamics, entrainment, mixing, turbulence, combustion and evaporating droplets. The macroscopic turbulent behaviour and thermodynamical properties of the model are obtained using DNS from the original governing equations.

DSTAR uses implicit compact finite difference schemes for spatial discretisation with sixth-, fourth- and third-order schemes for internal, near-boundary and boundary points, respectively. A fourth-order Lagrangian interpolation scheme is utilised to obtain gas properties at droplet locations. A third-order explicit Runge--Kutta scheme is used for temporal advancement of the flow variables, while a semi-analytical scheme is employed for droplet marching, with the consideration of numerical accuracy, stability and efficiency.

The work performed in this dCSE project will enable DSTAR to simulate new physical regimes over larger domains and longer times scales by increasing parallel scalability, serial performance and employing more efficient input/output operations for large data sets.

The overall objective was to achieve a DSTAR code that will scale to at least 10,000 XE6 cores with 80% efficiency and be capable to handle datasets for cubic grids of at least size 500. This would be achieved by the following:

  • Improve the scalability of the DSTAR multi-physics modules by implementing a 2D domain decomposition for the RHS calculations of the flow variables and derived quantities, including the spatial derivatives and global transpose operations.
  • Optimise the I/O operations which are used for monitoring the evolutionary physical quantities and restarts.
  • Re-factor the legacy Fortran 77 code by implementing MODULES in place of common blocks, employ the USE statement in place of INCLUDE and make more efficient use of available memory by using ALLOCATABLE arrays wherever possible.
  • Produce a brief user document for DSTAR users.

The upgraded version of DSTAR can now use approximately 50 times more MPI tasks and grid points than before and with good efficiency. The serial performance has also been improved significantly (approximately 20%). The parallel efficiency of the 2D decomposition, which takes approximately 90% of the total simulation run time has been improved by approximately 50%. The individual achievements of the project are summarised below:

  • A 2D domain decomposition was implemented by using the 2DECOMP&FFT library. Tests on two cubic grids of size 768 and 1536, using up to 18,432 MPI tasks achieved approximately 50% scaling efficiency on the XE6.
  • As an alternative to the 2D domain decomposition, a Mixed Mode programming model using a 1D domain decomposition was also prototyped. For large numbers of MPI processes DSTAR spends approximately 50% of the run time doing communication. It was found that better scalability can be achieved by using an algorithm that overlaps communication with computation by a Mixed Mode programming model. However to fully implement this method would result in a more complex source code.
  • DSTAR has two main I/O branches; one for monitoring which is in ASCII and ready for exporting for visualisation, and the other for restart data which is binary. The monitoring I/O is now performed by a dedicated process writing all the ASCII data rather than having many processes write their local data. An MPI-IO approach to writing the binary restart data was also implemented, however, the original method is still the fastest (i.e. where each node writes its' own data).
  • The Fortran 77 source code was re-factored and the main subroutines were re-developed as Fortran MODULES. Where subroutines made poor usage of cache memory utilisation, nested loops were reordered and any redundant IF THEN blocks were also removed, this gives approximately a further 20% increase in the performance of DSTAR.
  • For the user documentation, a README file was provided along with an appropriate makefile and directory structure. Recommendations for compilation with the main compilers on HECToR has also been provided.

For more information about these developments for DSTAR on HECToR, please see PDF or HTML for a report which summarises the project.