The HECToR Service is now closed and has been superceded by ARCHER.

Performance Enhancement and Optimization of the TPLS and DIM Two-Phase Flow Solvers

Understanding interfacial instabilities is one of the foremost challenging features of two-phase flows. These instabilities are of crucial practical and theoretical importance, including applications in oil/gas transport through pipelines, in process cleaning and in carbon-capture. The flows in these processes exhibit three-dimensional complexities on a wide range of spatial and temporal scales. The Two-Phase Level Set (TPLS) and Diffuse-Interface Method (DIM) codes are suites of two-phase solvers which may be used to model such flows. These applications solve the 3D Navier-Stokes equations, with a choice of interface tracking methods to describe the phase identifier.

The aim of this project is to enable TPLS and DIM to utilise HECToR for the simulation of flows on grids with at least 20 million points. The ability to model high-resolution geometries will then help improve our understanding of complex interfacial phenomena in two-phase flows. This will be achieved by the following:

  • Optimizing the MPI communications in TPLS by replacing the existing point-to-points (MPI_Sendrecv) with non-blocking alternatives. This should also enable more effective placement of OpenMP for improved mixed-mode performance.
  • Improving the convergence of the Pressure solver in TPLS. This will be achieved by replacing the existing successive over-relaxation pressure solver (for inverting Laplace's equation) with a hybrid OpenMP/MPI mixed-mode preconditioned conjugate-gradient routine.

The overall outcome of this work may be summarised as follows:

  • General improvements to the TPLS code were performed. A number of redundant MPI barriers and some OpenMP collective operations were removed in addition to some tuning of the OpenMP directives. Also, non-blocking point-to-point communications were introduced to allow multiple communications to be overlapped and therefore reduce any latency overhead. For a test case (OpenDuct1) with 512x144x152 grid points, a 12% speedup was achieved.
  • The most expensive area of computation in the TPLS code is the pressure calculation. The original method which used successive over-relaxation was replaced by calls to the PETSc 3.3 library which also allows use of various preconditioners and Krylov solvers.
  • An improvement in performance of up to 54% in the pressure solver has been achieved with 2048 cores for the OpenDuct1 case. This speedup will enable two-phase flow simulations at ultra-high resolution for a variety of engineering applications.
  • Excellent strong and weak scaling of the code was demonstrated with up to 1024x144x152 grid points (over 22 million points).
  • The developments from this project have been introduced within the TPLS code, which is available under the open-source BSD license from https://sourceforge.net/projects/tpls/.

Please see PDF or HTML for a report which summarises this project.