Through a dedicated Computational Science and Engineering project sponsored by the HECToR dCSE programme, the general purpose CFD application CABARET has been ported to run efficiently on the HECToR UK national supercomputer resource. Scalability of the CABARET application has been demonstrated. This work will now enable leading research to be performed on MILES modelling and general turbulence simulations and introduce a step change in the capability of the application.
In this report, we have described the development of a scalable hybrid parallel implementation of the CABARET application. This has taken place over the Phase 1 to Phase 2b (with Gemini) lifetime of HECToR. The CABARET method is particularly suitable for modelling high Reynolds number low Mach number turbulent flows in Large Eddy simulations. The method ensures that only nearest neighbour cells are involved and is therefore suited to distributed computing architectures.
The input grid is a Gambit(Fluent) produced unstructured hexahedral mesh and we have implemented an automated method of grid partitioning - using either a structured (cartesian plane) decomposition or unstructured external package - METIS/Scotch. Where the geometrical arrangement permits, users are recommended to employ a structured partitioning to ensure efficient load balancing. An MPI parallel version of CABARET was developed and tested on Phase 2a. The format of the main computational loops in CABARET make it difficult to achieve any form of vectorisation with any of the Fortran 90 compilers available on HECToR.
The arrival of Phase 2b and increasingly more multi-core focussed architecture brought the need for a hybrid parallel version of CABARET. For a representative size test case, it was shown that the performance of a pure MPI version of CABARET was not as good on the 24 core per node Phase 2b architecture as it had previously shown on the quad core nodes of Phase 2a. Implementing hybrid parallelism in the form of multi-threaded work sharing for the main loops has helped performance and enabled more efficiency of multi-core use of CABARET.
The overall outcome of this work may be summarised as follows:
In addition to meeting the original aims of the project, the follow has also been achieved:
The main users of CABARET are now able to investigate problem sizes at least 100 times larger than previously possible. Particular HECToR use of CABARET includes 7M AUs which was awarded at the November 2010 RAP for the project entitled ``Turbulent Flow Effects on Flap Noise''. A further HECToR resource of 21M AUs over Phase 2b and over 21M AUs for Phase 3, for further investigation in high Reynolds number turbulent flows of Jet Flap noise (EP/I017747/1) will use CABARET as the main code. CABARET is available to all HECToR users by contacting the HECToR CSE team (firstname.lastname@example.org).
Phil Ridley 2011-02-01