Developing the TELEMAC system for HECToR (phase 2b & beyond)
The TELEMAC-2D unstructured coastal modelling code is a new application on HECToR. TELEMAC uses the Finite Element Method to simulate free-surface flows, mainly environmental, dealing with current distribution, sediment transport, wave propagation, flooding and/or water quality treatment. The aim of this DCSE project was to improve the performance of TELEMAC on many core architectures by the use of OpenMP within TELEMAC's underlying finite element library, BIEF. Thus enabling increased usage of TELEMAC on HECToR. TELEMAC-2D is written in Fortran 90 and contains a total of 250,000 lines of code which is split into several libraries.
The overall aim of this project was to improve the scalability of TELEMAC-2D on many-core archiecturesre by:
- Developing OpenMP within the existing MPI for hybrid-parallelism for the linear solvers of the main finite element solver library BIEF and also the advection solver.
- Implementing OpenMP within the Conjugate Gradient (CG) solver and Positive Streamwise Invariant (PSI) scheme.
The achievements of the project are summarised below:
- OpenMP directives were implemented in the CG solver and in the Positive Streamwise Invariant scheme, in total 200 subroutines were updated. Benchmarking tests were demonstrated showing that a simulation using MPI / OpenMP (384 cores and 2 threads) in the Conjugate Gradient solver is now 80% faster than a simulation using pure MPI with the same number of cores on HECToR Phase 2b.
- It was not easy to implement OpenMP due to the indirect addressing used in many of the loops. This is because of the unstructured nature of the Finite Element Method.
- The reverse Cuthill-McKee (RCM) method was also implemented to re-order the mesh elements of the assembled finite element matrix and reduce bandwidth.
- This work has been made available to HECToR users of TELEMAC-2D.
Please see PDF for a report which summarises this work.
