Conclusion

Through a dedicated Computational Science and Engineering project sponsored by the HECToR dCSE programme, the general purpose CFD application CABARET has been ported to run efficiently on the HECToR UK national supercomputer resource. Scalability of the CABARET application has been demonstrated. This work will now enable leading research to be performed on MILES modelling and general turbulence simulations and introduce a step change in the capability of the application.

In this report, we have described the development of a scalable hybrid parallel implementation of the CABARET application. This has taken place over the Phase 1 to Phase 2b (with Gemini) lifetime of HECToR. The CABARET method is particularly suitable for modelling high Reynolds number low Mach number turbulent flows in Large Eddy simulations. The method ensures that only nearest neighbour cells are involved and is therefore suited to distributed computing architectures.

The input grid is a Gambit(Fluent) produced unstructured hexahedral mesh and we have implemented an automated method of grid partitioning - using either a structured (cartesian plane) decomposition or unstructured external package - METIS/Scotch. Where the geometrical arrangement permits, users are recommended to employ a structured partitioning to ensure efficient load balancing. An MPI parallel version of CABARET was developed and tested on Phase 2a. The format of the main computational loops in CABARET make it difficult to achieve any form of vectorisation with any of the Fortran 90 compilers available on HECToR.

The arrival of Phase 2b and increasingly more multi-core focussed architecture brought the need for a hybrid parallel version of CABARET. For a representative size test case, it was shown that the performance of a pure MPI version of CABARET was not as good on the 24 core per node Phase 2b architecture as it had previously shown on the quad core nodes of Phase 2a. Implementing hybrid parallelism in the form of multi-threaded work sharing for the main loops has helped performance and enabled more efficiency of multi-core use of CABARET.

The overall outcome of this work may be summarised as follows:

1. An automated geometrical domain decomposition method for partitioning a Gambit generated unstructured grid. The resulting partitions are ensured to give good load balancing for the main computation and minimise communication. The partitioner was demonstrated on a $5.12\times10^7$ grid and is currently capable of processing a $2.05\times10^8$ grid on HECToR Phase 2b. An internal report summarising methods for unstructured partitioning was written [13].
2. A parallel version of CABARET was implemented using non-blocking MPI to pass data between internal boundaries (cell faces and sides). This code was validated and tested against the old code using the 3D backward-facing step case and a parallel efficiency of 72% was observed when using 250 cores of the XT4 part of HECToR with the quad core Phase 2a architecture.
3. The new code was validated for a larger test case of $5.12\times10^7$ grid points on the XT4 part of HECToR, using the 3D backward-facing step case.

In addition to meeting the original aims of the project, the follow has also been achieved:

1. Implementation of a hybrid (OpenMP/MPI) parallel version of CABARET for Phase 2b and further. Also a hybrid parallel partitioner has been developed for the domain decomposition of the Gambit generated unstructured grid.
2. A 3D backward-facing step test case of $5.12\times10^7$ cells, retains 80% parallel efficiency on 1000 cores of Phase 2b.
3. Simulations can now be performed on grids at least 512 times larger than previously possible. Smaller scale simulations can now be performed in a few minutes that would otherwise have taken several days before this work.

The main users of CABARET are now able to investigate problem sizes at least 100 times larger than previously possible. Particular HECToR use of CABARET includes 7M AUs which was awarded at the November 2010 RAP for the project entitled ``Turbulent Flow Effects on Flap Noise''. A further HECToR resource of 21M AUs over Phase 2b and over 21M AUs for Phase 3, for further investigation in high Reynolds number turbulent flows of Jet Flap noise (EP/I017747/1) will use CABARET as the main code. CABARET is available to all HECToR users by contacting the HECToR CSE team (support@hector.ac.uk).